Downloading files from FTP Server where directory changed dynamically - spring

I am using Spring Integration to download/upload files from FTP server.
How can I change remote-directory="/directory Name" dynamically in Spring FTP:Inbound-Channel.
My client will create a folder daily basically in "MM-dd-yy" format and copy all files there.
In "FTP:Inbound-channel" I did not find any way to configure this pattern. I basically have
to hardcord the directory or file names in configuration.
What I want is to set the path programatically. Because some times I need to download all files
from a direcotory or download only a specific file.
I found "remote-directory-expression="'directory'+'/'+ new java.text.SimpleDateFormat('dd-MM-yyyy').format(new java.util.Date())" can be set in FTP:Outbound-channel
is there any such attribute in FTP:InBound-channel
My configuration is like this:
<bean id="ftpClientFactory"
class="org.springframework.integration.ftp.session.DefaultFtpSessionFactory">
<property name="host" value="${host}" />
<property name="port" value="${availableServerPort}" />
<property name="username" value="${userid}" />
<property name="password" value="${password}" />
</bean>
<int-ftp:inbound-channel-adapter id="ftpInbound"
cache-sessions="false" channel="ftpChannel" session-factory="ftpClientFactory"
filename-pattern="*.txt" auto-create-local-directory="true"
delete-remote-files="false" remote-directory="/filedirectory"
local-directory="${local_directory}">
<int:poller fixed-rate="1000" />
</int-ftp:inbound-channel-adapter>
<int:channel id="ftpChannel">
<int:queue />
</int:channel>
I did not find a way to do all the above items.
Please let me know how can I achieve this.

You can't do it with the inbound adapter, but the <ftp:outbound-gateway/> can be used to achieve what you need; described here.
You can either use ls to list the files, followed a <splitter/> and another gateway using get; or you can use the mget command with a file name pattern in the expression.
The FTP sample has an example of using the gateway

Related

Transfer files on different servers (sftp) using Spring Integration

I have a requirement to build an application in SI which reads an input directory which may consist 1000s of file and copy them to remote servers, say 10 servers where a processor instance will pick them up for processing.The movement of file should be on round-robin fashion so that there is not additional burden on any server while processing them. To elaborate a little more - lets say we have 10 files in input directory then application should copy
file 1 on server1,
file2 on server2
.
.
.
....
file 10 on server 10.
Sequence doesn't matter what matters is that every server should have equal load.I am fairly new to Spring Integration but I found a sample to do sftp of file using SI
https://github.com/spring-projects/spring-integration-samples/tree/master/basic/sftp
but i am not sure how can I configure it for multiple servers and to have an algo to move files in round-robin fashion.
I will appreciate any tips or suggestion .
I am able to do sftp using below config.
<context:property-placeholder location="classpath:app.properties" />
<int-file:inbound-channel-adapter id="ReaderChannel"
directory="file:${input.file.dir}" filename-pattern="*.*"
prevent-duplicates="true" ignore-hidden="true" auto-startup="true">
<int:poller id="poller" fixed-rate="1" task-executor="myTaskExecutor" />
</int-file:inbound-channel-adapter>
<int-task:executor id="myTaskExecutor" pool-size="${file.concurrentFilesNum}" queue-capacity="0" rejection-policy="CALLER_RUNS" />
<int-sftp:outbound-channel-adapter id="sftpOutboundAdapter" session-factory="sftpSessionFactory" channel="ReaderChannel"
charset="UTF-8" remote-directory="${output.file.dir}" auto-startup="true">
<int-sftp:request-handler-advice-chain>
<int:retry-advice />
</int-sftp:request-handler-advice-chain>
</int-sftp:outbound-channel-adapter>
<beans:bean id="sftpSessionFactory" class="org.springframework.integration.file.remote.session.CachingSessionFactory">
<beans:constructor-arg ref="defaultSftpSessionFactory" />
</beans:bean>
<beans:bean id="defaultSftpSessionFactory"
class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<beans:property name="host" value="${sftp.host}" />
<beans:property name="privateKey" value="${sftp.private.keyfile}" />
<beans:property name="privateKeyPassphrase" value="${sftp.private.passphrase}" />
<beans:property name="port" value="${sftp.serverPort}" />
<beans:property name="user" value="${sftp.username}" />
<beans:property name="allowUnknownKeys" value="true" />
</beans:bean>
The round-robin is hidden in the DirectChannel with UnicastingDispatcher on the RoundRobinLoadBalancingStrategy.
So, when you have several subscriber to the same DirectChannel, the message will be dispatched to them in the round-robin.
What you need for your use-case is just configure 10 <int-sftp:outbound-channel-adapter> for each your remote server. And use the same simple <channel> definition for their channel attribute.
The <int-file:inbound-channel-adapter> should always send its message to that shared channel with default round-robin strategy.

SFTP Spring Integrations rm command

I am wondering if someone can assist, I know I am doing this wrong and I'm tearing my hair out. My goal is to delete any files with a .txt extension in a remote directory using Spring Integrations SFTP in a Spring Batch job. It is my understanding that I do not have to ls remote files to remove them and can just issue an rm command on *.txt for a given directory however I may be incorrect?
I have the following SFTP configuration
<bean id="sftpSessionFactory"
class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<property name="host" value="${host}" />
<property name="port" value="${port}" />
<property name="user" value="${user}" />
<property name="privateKey" value="file:${privateKey}" />
<property name="password" value="${password}" />
</bean>
<int:channel id="rmChannel"/>
<int-sftp:outbound-gateway
session-factory="sftpSessionFactory"
request-channel="rmChannel"
remote-file-separator="/"
command="rm"
expression="headers['file_remoteDirectory'] + headers['file_remoteFile']" />
<bean id="cleanRemoteDirectoryTasklet" class="com.example.batch.job.integration.SFTPRmTasklet" scope="step">
<property name="channel" ref="rmChannel" />
<property name="filePatternToDelete" value="*.txt" />
<property name="targetDirectory" value="${remoteDirectory}"/> // edit removed 'file:' notation
</bean>
I believe I am OK to this point an my problem is executing this flow in the Java implementation in SFTPRmTasklet, I'm not sure how to construct the message to initiate the sftp remove. Currently I have something like this simply to kick it off I know that my payload is wrong.
Message<String> rmRequest = MessageBuilder.withPayload("/my/target/dir")
.setHeader("file_remoteDirectory", targetDirectory)
.setHeader("file_remoteFile", filePatternToDelete)
.build();
channel.send(rmRequest)
ultimately this yields an exception
org.springframework.integration.MessagingException: org.springframework.core.NestedIOException: Failed to remove file: 2: No such file
UPDATE 1
So I decided to target just one remote file and changed filePatternToDelete to test.txt. After a bit of debugging I realised that AbstractRemoteFileOutBoundGateway was evaluating my remoteFilePath to /my/target/dirtest.txt and remote filename to dirtest.txt, which is obviously not what I wanted so I added a trailing to / to the target directory in my properties file and this sorted out this error great!
I can now delete the file from the remote server as I wished to do however I received an error around no reply-channel so I have added the following channel
<int:channel id="end"/>
and modified my outbound gateway
<int-sftp:outbound-gateway
session-factory="sftpSessionFactory"
request-channel="rmChannel"
reply-channel="end"
remote-file-separator="/"
command="rm"
expression="headers['file_remoteDirectory'] + headers['file_remoteFile']" />
and now get an error around no subscribers for this channel. Progress at least and in case you hadn't guessed I'm pretty new to Spring!
If you are not interested in the result of the rm, set the reply-channel to nullChannel.
It's like /dev/null on unix.

how to trigger many ftp inbound-channel-adapter from one poller to connect ftp server?

First, thank you for your attention
i defined two ftp inbound-channel-adapter in my spring integration project,the adapters with diferent configuration but have share session factory to connect ftp server, each adapters have one poller to trigger,i want to see the bellow code is correct?
is efficiency?
<bean id="ftpClientFactory"
class="org.springframework.integration.ftp.session.DefaultFtpSessionFactory">
<property name="host" value="127.0.0.1"/>
<property name="port" value="21"/>
<property name="username" value="banks_reader"/>
<property name="password" value="123456"/>
</bean>
<bean id="myFilter" class="com.ali.util.FtpFilter"/>
<bean id="onceFilter" class="com.ali.util.OnceFilter"/>
<int-ftp:inbound-channel-adapter id="ftpInbound1"
channel="inboundFileChannel"
**session-factory="ftpClientFactory"**
charset="UTF-8"
auto-create-local-directory="true"
delete-remote-files="true"
remote-directory="/directoryA"
remote-file-separator="/"
temporary-file-suffix=".writing"
local-filter="myFilter"
filter="onceFilter"
local-directory="file:output">
**<int:poller fixed-rate="5000"/>**
</int-ftp:inbound-channel-adapter>
<int-ftp:inbound-channel-adapter id="ftpInbound2"
channel="inboundFileChannel"
**session-factory="ftpClientFactory"**
charset="UTF-8"
auto-create-local-directory="true"
delete-remote-files="true"
remote-directory="/directoryB"
remote-file-separator="/"
temporary-file-suffix=".writing"
local-filter="myFilter"
filter="onceFilter"
local-directory="file:output">
**<int:poller fixed-rate="5000"/>**
</int-ftp:inbound-channel-adapter>
Is there another way?
That is the correct technique using channel adapters.
Or, you could use an outbound gateway instead, and fetch (ls + get or mget) files from both directories in turn.

How to receive details of file on ftp server using spring-integration?

I've been searching for answer but I didn't find any. How can I receive information about, for example date of creation file stored on ftp server using spring-integration? Is any chance to do this using inbound-channel-adapter or something like that?
Thanks for advance.
#EDIT:
Configuration:
<int:splitter id="splitter" input-channel="toSplitter" output-channel="ftpChannel"/>
<int-ftp:outbound-gateway id="gatewayLS"
session-factory="ftpClientFactory"
request-channel="inbound"
command="ls"
expression="payload"
reply-channel="toSplitter"/>
<int:channel id="toSplitter">
<int:interceptors>
<int:wire-tap channel="logger"/>
</int:interceptors>
</int:channel>
<bean id="ftpClientFactory"
class="org.springframework.integration.ftp.session.DefaultFtpSessionFactory">
<property name="host" value="${host}"/>
<property name="username" value="${user}"/>
<property name="password" value="${password}"/>
<property name="clientMode" value="0"/>
<property name="fileType" value="2"/>
<property name="bufferSize" value="10000000"/>
</bean>
<int:channel id="ftpChannel"/>
#EDIT:
Problem is solved thanks to Gary.
Use the ls command of the outbound gateway.
As stated there: "The message payload resulting from an ls operation is a list of file names, or a list of FileInfo objects. These objects provide information such as modified time, permissions etc."
Also, see the FTP Sample App.

Uploading files directly to specific folder in ftp server using spring spring

I have, I suppose, really newbie question but the fact is I'm newbie in spring framework.
How can I upload files to, for example 'upload' folder loceted in root directory of my ftp server?
I have tried this:
My application context file:
<bean id="ftpClientFactory"
class="org.springframework.integration.ftp.session.DefaultFtpSessionFactory">
<property name="host" value="127.0.0.1"/>
<property name="username" value="test"/>
<property name="password" value="test"/>
<property name="clientMode" value="0"/>
<property name="fileType" value="2"/>
<property name="bufferSize" value="10000"/>
</bean>
<int:channel id="ftpChannel"/>
<int-ftp:outbound-channel-adapter id="outFtpAdapter"
channel="ftpChannel"
session-factory="ftpClientFactory"
remote-directory="/Users/test"/>
and my java code:
ConfigurableApplicationContext context =
new FileSystemXmlApplicationContext("/src/citrus/resources/citrus-context.xml");
MessageChannel ftpChannel = context.getBean("ftpChannel", MessageChannel.class);
File file = new File("/Users/test/test.txt");
Message<File> fileMessage = MessageBuilder.withPayload(file).build();
ftpChannel.send(fileMessage);
context.close();
But this example upload files to root directory.
Thanks in advance.
I've just tested it and work well:
<int-ftp:outbound-channel-adapter
id="sendFileToServer"
auto-create-directory="true"
session-factory="ftpSessionFactory"
remote-directory="/Users/test"/>
Where my FTP server is an embedded one and its root is:
[MY_USER_HOME]\Temp\junit7944189423444999123\FtpServerOutboundTests\
So the file is stored in the dir:
[MY_USER_HOME]\Temp\junit7944189423444999123\FtpServerOutboundTests\Users\test\
It is with the latest Spring Integration version.
Which is your version?
Using FTPClient upload MultipartFiles to apache FTP server
useful repo
Hope this help for future search.

Resources