Spring Integration ftp inboud channel processing files in specific order - spring

I'm using Spring Integration to poll files from remote FTP server and process them.
Is there a way to configure FtpInboundFileSynchronizer (or other component) to fetch and process remote files in specific order. Say i have file1 and file2 in remote directory, is it possible to fetch and process file1 before file2.
Thanks in advance

There are (at least) 3 techniques to achieve this:
Add a custom FileListFilter<FTPFile> (that sorts the FTPFile objects into the order you desire) to the synchronizer.
Use two FTP outbound gateways, one the list (ls) the files, and one to get each file as needed.
Use the FtpRemoteFileTemplate from within your own code to list and fetch files.
EDIT
Actually, for #1, you would also need a custom FileListFilter<File> in the local filter to sort the File objects. Since the local files are emitted as message payloads after the synchronization is complete.
EDIT2 Remote file template example
This just copies the first file in the list, but it should give you what you need...
#SpringBootApplication
public class So49462148Application {
public static void main(String[] args) {
SpringApplication.run(So49462148Application.class, args);
}
#Bean
public ApplicationRunner runner(FtpRemoteFileTemplate template) {
return args -> {
FTPFile[] files = template.list("*.txt");
System.out.println(Arrays.toString(files));
template.get(files[0].getName(), is -> {
File file = new File("/tmp/" + files[0].getName());
FileOutputStream os = new FileOutputStream(file);
FileCopyUtils.copy(is, os);
System.out.println("Copied: " + file.getAbsolutePath());
});
};
}
#Bean
public DefaultFtpSessionFactory sf() {
DefaultFtpSessionFactory sf = new DefaultFtpSessionFactory();
sf.setHost("...");
sf.setUsername("...");
sf.setPassword("...");
return sf;
}
#Bean
public FtpRemoteFileTemplate template(DefaultFtpSessionFactory sf) {
FtpRemoteFileTemplate template = new FtpRemoteFileTemplate(sf);
template.setRemoteDirectoryExpression(new LiteralExpression("foo"));
return template;
}
}

Related

Spring SFTP Outbound Adapter - determining when files have been sent

I have a Spring SFTP output adapter that I start via "adapter.start()" in my main program. Once started, the adapter transfers and uploads all the files in the specified directory as expected. But I want to stop the adapter after all the files have been transferred. How do I detect if all the files have been transferred so I can issue an adapter.stop()?
#Bean
public IntegrationFlow sftpOutboundFlow() {
return IntegrationFlows.from(Files.inboundAdapter(new File(sftpOutboundDirectory))
.filterExpression("name.endsWith('.pdf') OR name.endsWith('.PDF')")
.preventDuplicates(true),
e -> e.id("sftpOutboundAdapter")
.autoStartup(false)
.poller(Pollers.trigger(new FireOnceTrigger())
.maxMessagesPerPoll(-1)))
.log(LoggingHandler.Level.INFO, "sftp.outbound", m -> m.getPayload())
.log(LoggingHandler.Level.INFO, "sftp.outbound", m -> m.getHeaders())
.handle(Sftp.outboundAdapter(outboundSftpSessionFactory())
.useTemporaryFileName(false)
.remoteDirectory(sftpRemoteDirectory))
.get();
}
#Artem Bilan has already given the answer. But here's kind of a concrete implementation of what he said - for those who are a Spring Integration noob like me:
Define a service to get the PDF files on demand:
#Service
public class MyFileService {
public List<File> getPdfFiles(final String srcDir) {
File[] files = new File(srcDir).listFiles((dir, name) -> name.toLowerCase().endsWith(".pdf"));
return Arrays.asList(files == null ? new File[]{} : files);
}
}
Define a Gateway to start the SFTP upload flow on demand:
#MessagingGateway
public interface SFtpOutboundGateway {
#Gateway(requestChannel = "sftpOutboundFlow.input")
void uploadFiles(List<File> files);
}
Define the Integration Flow to upload the files to the SFTP server via Sftp.outboundGateway:
#Configuration
#EnableIntegration
public class FtpFlowIntegrationConfig {
// could be also bound via #Value
private String sftpRemoteDirectory = "/path/to/remote/dir";
#Bean
public SessionFactory<ChannelSftp.LsEntry> outboundSftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost("localhost");
factory.setPort(22222);
factory.setUser("client1");
factory.setPassword("password123");
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<>(factory);
}
#Bean
public IntegrationFlow sftpOutboundFlow(RemoteFileTemplate<ChannelSftp.LsEntry> remoteFileTemplate) {
return e -> e
.log(LoggingHandler.Level.INFO, "sftp.outbound", Message::getPayload)
.log(LoggingHandler.Level.INFO, "sftp.outbound", Message::getHeaders)
.handle(
Sftp.outboundGateway(remoteFileTemplate, AbstractRemoteFileOutboundGateway.Command.MPUT, "payload")
);
}
#Bean
public RemoteFileTemplate<ChannelSftp.LsEntry> remoteFileTemplate(SessionFactory<ChannelSftp.LsEntry> outboundSftpSessionFactory) {
RemoteFileTemplate<ChannelSftp.LsEntry> template = new SftpRemoteFileTemplate(outboundSftpSessionFactory);
template.setRemoteDirectoryExpression(new LiteralExpression(sftpRemoteDirectory));
template.setAutoCreateDirectory(true);
template.afterPropertiesSet();
template.setUseTemporaryFileName(false);
return template;
}
}
Wiring up:
public class SpringApp {
public static void main(String[] args) {
final MyFileService fileService = ctx.getBean(MyFileService.class);
final SFtpOutboundGateway sFtpOutboundGateway = ctx.getBean(SFtpOutboundGateway.class);
// trigger the sftp upload flow manually - only once
sFtpOutboundGateway.uploadFiles(fileService.getPdfFiles());
}
}
Import notes:
1.
#Gateway(requestChannel = "sftpOutboundFlow.input")
void uploadFiles(List files);
Here the DirectChannel channel sftpOutboundFlow.input will be used to pass message with the payload (= List<File> files) to the receiver. If this channel is not created yet, the Gateway is going to create it implicitly.
2.
#Bean
public IntegrationFlow sftpOutboundFlow(RemoteFileTemplate<ChannelSftp.LsEntry> remoteFileTemplate) { ... }
Since IntegrationFlow is a Consumer functional interface, we can simplify the flow a little using the IntegrationFlowDefinition. During the bean registration phase, the IntegrationFlowBeanPostProcessor converts this inline (Lambda) IntegrationFlow to a StandardIntegrationFlow and processes its components. An IntegrationFlow definition using a Lambda populates DirectChannel as an inputChannel of the flow and it is registered in the application context as a bean with the name sftpOutboundFlow.input in the sample above (flow bean name + ".input"). That's why we use that name for the SFtpOutboundGateway gateway.
Ref: https://spring.io/blog/2014/11/25/spring-integration-java-dsl-line-by-line-tutorial
3.
#Bean
public RemoteFileTemplate<ChannelSftp.LsEntry> remoteFileTemplate(SessionFactory<ChannelSftp.LsEntry> outboundSftpSessionFactory) {}
see: Remote directory for sftp outbound gateway with DSL
Flowchart:
But I want to stop the adapter after all the files have been transferred.
Logically this is not for what this kind of component has been designed. Since you are not going to have some constantly changing local directory, probably it is better to think about an even driver solution to list files in the directory via some action. Yes, it can be a call from the main, but only once for all the content of the dir and that's all.
And for this reason the Sftp.outboundGateway() with a Command.MPUT is there for you:
https://docs.spring.io/spring-integration/reference/html/sftp.html#using-the-mput-command.
You still can trigger an IntegrationFlow, but it could start from a #MessagingGateway interface to be called from a main with a local directory to list files for uploading:
https://docs.spring.io/spring-integration/reference/html/dsl.html#java-dsl-gateway

Spring Integration File synchronizer : Accept files based on a pre defined list

I am transfering files from a remote to a local by Sftp, for processing.
I want to only transfer .csv files, and I have a list of pre-defined filenames.
I couldn't find a FileListFilter that allows to specify multiple patterns and transfer if at least one if matched.
So far I have this code, that's woorking for ".csv" filtering.
The Integration Flow
#Bean
public IntegrationFlow integFlow() {
return IntegrationFlows
.from(ftpMessageSource(), c -> poller())
... more processing
The MessageSource
public MessageSource<File> ftpMessageSource() {
SftpInboundFileSynchronizer fileSynchronizer = new SftpInboundFileSynchronizer(sessionFactory);
fileSynchronizer.setRemoteDirectory(remoteDirectory);
fileSynchronizer.setDeleteRemoteFiles(true);
fileSynchronizer.setFilter(new SftpRegexPatternFileListFilter(Constantes.EXTENSION));
SftpInboundFileSynchronizingMessageSource ftpInboundFileSync =
new SftpInboundFileSynchronizingMessageSource(fileSynchronizer);
ftpInboundFileSync.setLocalDirectory(new File(workDirectory));
ftpInboundFileSync.setAutoCreateLocalDirectory(true);
CompositeFileListFilter<File> compositeFileListFilter = new CompositeFileListFilter<>();
compositeFileListFilter.addFilter(new RegexPatternFileListFilter(Constantes.EXTENSION));
ftpInboundFileSync.setLocalFilter(compositeFileListFilter);
return ftpInboundFileSync;
}
Constantes.EXTENSION is a regex accepting .csv and .CSV. This works fine.
Say that I have a String list that contains "string1',"string2","string3" and I want to transfer every file of the form string1*, string2* or string3*. How would I proceed ?
#SpringBootApplication
public class So59161698Application {
public static void main(String[] args) {
SpringApplication.run(So59161698Application.class, args);
}
private final String myPatterns = "foo,bar,baz";
#Bean
public FileListFilter<File> filter() {
Set<String> patterns = StringUtils.commaDelimitedListToSet(this.myPatterns);
return files -> Arrays.stream(files)
.filter(file -> patterns.stream()
.filter(pattern -> file.getName().startsWith(pattern))
.findFirst()
.isPresent())
.collect(Collectors.toList());
}
#Bean
public ApplicationRunner runner(FileListFilter<File> filter) {
return args -> {
System.out.println(filter.filterFiles(new File[] {
new File("foo.csv"),
new File("bar.csv"),
new File("baz.csv"),
new File("qux.csv")
}));
};
}
}
[foo.csv, bar.csv, baz.csv]
There is a CompositeFileListFilter:
* Simple {#link FileListFilter} that predicates its matches against <b>all</b> of the
* configured {#link FileListFilter}.
With the logic like:
public boolean accept(F file) {
AtomicBoolean allAccept = new AtomicBoolean(true);
// we can't use stream().allMatch() because we have to call all filters for this filter's contract
this.fileFilters.forEach(f -> allAccept.compareAndSet(true, f.accept(file)));
return allAccept.get();
}
So, you configure this CompositeFileListFilter with several SftpRegexPatternFileListFilter delegates and your files are going to be processed whenever they much at least one of the filters in the CompositeFileListFilter.
See more about filters in the docs: https://docs.spring.io/spring-integration/docs/current/reference/html/file.html#file-reading

Spring Batch: FlatFileItemWriter to InputSteam/Byte array

So I've created a batch job which generates reports (csv files). I have been able to generate the the files seamlessly using FlatFileItemWriter but my end goal is to create an InputStream to call a rest service which will store the document or a byte array to store it in the database.
public class CustomWriter implements ItemWriter<Report> {
#Override
public void write(List<? extends Report> reports) throws Exception {
reports.forEach(this::writeDataToFile);
}
private void writeDataToFile(final Report data) throws Exception {
FlatFileItemWriter writer = new FlatFileItemWriter();
writer.setResource(new FileSystemResource("C:/reports/test-report.csv"));
writer.setLineAggregator(getLineAggregator();
writer.afterPropertiesSet();
writer.open(new ExecutionContext());
writer.write(data);
writer.close();
}
private DelimitedLineAggregator<Report> getLineAggregator(final Report report) {
DelimitedLineAggregator<Report> delimitedLineAgg = new DelimitedLineAggregator<Report>();
delimitedLineAgg.setDelimiter(",");
delimitedLineAgg.setFieldExtractor(getFieldExtractor());
return delimitedLineAgg;
}
private FieldExtractor<Report> getFieldExtractor() {
BeanWrapperFieldExtractor<Report> fieldExtractor = new BeanWrapperFieldExtractor<Report>();
fieldExtractor.setNames(COLUMN_HEADERS.toArray(new String[0]));
return fieldExtractor;
}
}
One way I could do this is to store the file locally temporarily and create a new step to pick the generated files up and do the sending/storing but I would really like to skip this step and send/store it in the first step.
How do I go about doing this?

Discussion about spring integration sftp

I use spring integration sftp to download and upload files.In the document ,I found
Spring Integration supports sending and receiving files over SFTP by providing three client side endpoints: Inbound Channel Adapter, Outbound Channel Adapter, and Outbound Gateway
When I want to download files I must assign the local directory and when I want to upload files I must assign the remote directory.But if I can't assign the directory when I write the code such as my directory is association with date.How can I assign the directory at runtime?
Here is my code:
#Bean
public SessionFactory<LsEntry> sftpSessionFactory(){
DefaultSftpSessionFactory defaultSftpSessionFactory = new DefaultSftpSessionFactory();
defaultSftpSessionFactory.setHost(host);
defaultSftpSessionFactory.setPort(Integer.parseInt(port));
defaultSftpSessionFactory.setUser(username);
defaultSftpSessionFactory.setPassword(password);
defaultSftpSessionFactory.setAllowUnknownKeys(true);
return new CachingSessionFactory<LsEntry>(defaultSftpSessionFactory);
}
#Bean
public SftpRemoteFileTemplate sftpRemoteFileTemplate(){
SftpRemoteFileTemplate sftpRemoteFileTemplate = new SftpRemoteFileTemplate(sftpSessionFactory());
return sftpRemoteFileTemplate;
}
#Bean
#ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler handlerGet() {
SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(sftpSessionFactory(), "mget", "payload");
sftpOutboundGateway.setLocalDirectory(new File(localDirectory));
sftpOutboundGateway.setFilter(new SftpSimplePatternFileListFilter("*.txt"));
sftpOutboundGateway.setSendTimeout(1000);
return sftpOutboundGateway;
}
In the messageHandler,I must assign the localDirectory in the outboundGateway. And when I want change my localDirectory by days.I must download the file to the localDirectory and move to the target directory. How can I assign the localDirectory at runtime .such as today I download to 20170606/ and tomorrow I download to 20170607 ?
edit
this is my option and test
public interface OutboundGatewayOption {
#Gateway(requestChannel = "sftpChannel")
public List<File> getFiles(String dir);
}
#Test
public void test2(){
outboundGatewayOption.getFiles("upload/20160920/");
}
sftpOutboundGateway.setLocalDirectoryExpression(
new SpelExpressionParser().parseExpression("headers['whereToPutTheFiles']");
or parseExpression("#someBean.getDirectoryName(payload)")
etc.
The expression must evaluate to a String representing the directory absolute path.
While evaluating the expression, the remote directory is available as a variable #remoteDirectory.

Send and receive files from FTP in Spring Boot

I'm new to Spring Framework and, indeed, I'm learning and using Spring Boot. Recently, in the app I'm developing, I made Quartz Scheduler work, and now I want to make Spring Integration work there: FTP connection to a server to write and read files from.
What I want is really simple (as I've been able to do so in a previous java application). I've got two Quartz Jobs scheduled to fired in different times daily: one of them reads a file from a FTP server and another one writes a file to a FTP server.
I'll detail what I've developed so far.
#SpringBootApplication
#ImportResource("classpath:ws-config.xml")
#EnableIntegration
#EnableScheduling
public class MyApp extends SpringBootServletInitializer {
#Autowired
private Configuration configuration;
//...
#Bean
public DefaultFtpsSessionFactory myFtpsSessionFactory(){
DefaultFtpsSessionFactory sess = new DefaultFtpsSessionFactory();
Ftp ftp = configuration.getFtp();
sess.setHost(ftp.getServer());
sess.setPort(ftp.getPort());
sess.setUsername(ftp.getUsername());
sess.setPassword(ftp.getPassword());
return sess;
}
}
The following class I've named it as a FtpGateway, as follows:
#Component
public class FtpGateway {
#Autowired
private DefaultFtpsSessionFactory sess;
public void sendFile(){
// todo
}
public void readFile(){
// todo
}
}
I'm reading this documentation to learn to do so. Spring Integration's FTP seems to be event driven, so I don't know how can I execute either of the sendFile() and readFile() from by Jobs when the trigger is fired at an exact time.
The documentation tells me something about using Inbound Channel Adapter (to read files from a FTP?), Outbound Channel Adapter (to write files to a FTP?) and Outbound Gateway (to do what?):
Spring Integration supports sending and receiving files over FTP/FTPS by providing three client side endpoints: Inbound Channel Adapter, Outbound Channel Adapter, and Outbound Gateway. It also provides convenient namespace-based configuration options for defining these client components.
So, I haven't got it clear as how to follow.
Please, could anybody give me a hint?
Thank you!
EDIT:
Thank you #M. Deinum. First, I'll try a simple task: read a file from the FTP, the poller will run every 5 seconds. This is what I've added:
#Bean
public FtpInboundFileSynchronizer ftpInboundFileSynchronizer() {
FtpInboundFileSynchronizer fileSynchronizer = new FtpInboundFileSynchronizer(myFtpsSessionFactory());
fileSynchronizer.setDeleteRemoteFiles(false);
fileSynchronizer.setPreserveTimestamp(true);
fileSynchronizer.setRemoteDirectory("/Entrada");
fileSynchronizer.setFilter(new FtpSimplePatternFileListFilter("*.csv"));
return fileSynchronizer;
}
#Bean
#InboundChannelAdapter(channel = "ftpChannel", poller = #Poller(fixedDelay = "5000"))
public MessageSource<File> ftpMessageSource() {
FtpInboundFileSynchronizingMessageSource source = new FtpInboundFileSynchronizingMessageSource(inbound);
source.setLocalDirectory(new File(configuracion.getDirFicherosDescargados()));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new AcceptOnceFileListFilter<File>());
return source;
}
#Bean
#ServiceActivator(inputChannel = "ftpChannel")
public MessageHandler handler() {
return new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
Object payload = message.getPayload();
if(payload instanceof File){
File f = (File) payload;
System.out.println(f.getName());
}else{
System.out.println(message.getPayload());
}
}
};
}
Then, when the app is running, I put a new csv file intro "Entrada" remote folder, but the handler() method isn't run after 5 seconds... I'm doing something wrong?
Please add #Scheduled(fixedDelay = 5000) over your poller method.
You should use SPRING BATCH with tasklet. It is far easier to configure bean, crone time, input source with existing interfaces provided by Spring.
https://www.baeldung.com/introduction-to-spring-batch
Above example is annotation and xml based both, you can use either.
Other benefit Take use of listeners and parallel steps. This framework can be used in Reader - Processor - Writer manner as well.

Resources