I am wondering if someone can assist, I know I am doing this wrong and I'm tearing my hair out. My goal is to delete any files with a .txt extension in a remote directory using Spring Integrations SFTP in a Spring Batch job. It is my understanding that I do not have to ls remote files to remove them and can just issue an rm command on *.txt for a given directory however I may be incorrect?
I have the following SFTP configuration
<bean id="sftpSessionFactory"
class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<property name="host" value="${host}" />
<property name="port" value="${port}" />
<property name="user" value="${user}" />
<property name="privateKey" value="file:${privateKey}" />
<property name="password" value="${password}" />
</bean>
<int:channel id="rmChannel"/>
<int-sftp:outbound-gateway
session-factory="sftpSessionFactory"
request-channel="rmChannel"
remote-file-separator="/"
command="rm"
expression="headers['file_remoteDirectory'] + headers['file_remoteFile']" />
<bean id="cleanRemoteDirectoryTasklet" class="com.example.batch.job.integration.SFTPRmTasklet" scope="step">
<property name="channel" ref="rmChannel" />
<property name="filePatternToDelete" value="*.txt" />
<property name="targetDirectory" value="${remoteDirectory}"/> // edit removed 'file:' notation
</bean>
I believe I am OK to this point an my problem is executing this flow in the Java implementation in SFTPRmTasklet, I'm not sure how to construct the message to initiate the sftp remove. Currently I have something like this simply to kick it off I know that my payload is wrong.
Message<String> rmRequest = MessageBuilder.withPayload("/my/target/dir")
.setHeader("file_remoteDirectory", targetDirectory)
.setHeader("file_remoteFile", filePatternToDelete)
.build();
channel.send(rmRequest)
ultimately this yields an exception
org.springframework.integration.MessagingException: org.springframework.core.NestedIOException: Failed to remove file: 2: No such file
UPDATE 1
So I decided to target just one remote file and changed filePatternToDelete to test.txt. After a bit of debugging I realised that AbstractRemoteFileOutBoundGateway was evaluating my remoteFilePath to /my/target/dirtest.txt and remote filename to dirtest.txt, which is obviously not what I wanted so I added a trailing to / to the target directory in my properties file and this sorted out this error great!
I can now delete the file from the remote server as I wished to do however I received an error around no reply-channel so I have added the following channel
<int:channel id="end"/>
and modified my outbound gateway
<int-sftp:outbound-gateway
session-factory="sftpSessionFactory"
request-channel="rmChannel"
reply-channel="end"
remote-file-separator="/"
command="rm"
expression="headers['file_remoteDirectory'] + headers['file_remoteFile']" />
and now get an error around no subscribers for this channel. Progress at least and in case you hadn't guessed I'm pretty new to Spring!
If you are not interested in the result of the rm, set the reply-channel to nullChannel.
It's like /dev/null on unix.
Related
Versions:
Spring: 5.2.16.RELEASE
Spring Integrations: 5.3.9.RELEASE
macOS Big Sur: 11.6
I am using XML to set up the directory scanner FileReadingMessageSource.WatchServiceDirectoryScanner like so:
<int-file:inbound-channel-adapter id="channelIn" directory="${channel.dir}" auto-create-directory="false" use-watch-service="true" filter="channelFilter" watch-events="CREATE,MODIFY">
<int-file:nio-locker ref="channelLocker"/>
<int:poller fixed-delay="${channel.polling.delay}" max-messages-per-poll="${channel.polling.maxmsgs}"></int:poller>
</int-file:inbound-channel-adapter>
with the following bean definitions:
<bean id="channelLocker" class="org.springframework.integration.file.locking.NioFileLocker"/>
<bean id="channelFilter" class="org.springframework.integration.file.filters.ChainFileListFilter">
<constructor-arg>
<list>
<bean class="org.springframework.integration.file.filters.SimplePatternFileListFilter">
<constructor-arg value="SpreadSheets*.xls" />
</bean>
<bean id="filter" class="org.springframework.integration.file.filters.LastModifiedFileListFilter">
<property name="age" value="${channel.filter.age}" />
</bean>
<ref bean="persistentFilter" />
</list>
</constructor-arg>
</bean>
<bean id="persistentFilter" class="org.springframework.integration.file.filters.FileSystemPersistentAcceptOnceFileListFilter">
<constructor-arg index="0" ref="metadataStore" />
<constructor-arg index="1" name="prefix" value="" />
<property name="flushOnUpdate" value="false" />
</bean>
If I look at logs for org.springframework.integration.file.FileReadingMessageSource, I notice that we register both the specified directory (ie, ${channel.dir}) as well as any of its sub-directories. That is, I see logs like this:
15:44:45.706 [main] DEBUG org.springframework.integration.file.FileReadingMessageSource - registering: /Users/kc/scan.here for file events
15:44:45.711 [main] DEBUG org.springframework.integration.file.FileReadingMessageSource - registering: /Users/kc/scan.here/and.here for file events
I've looked Spring docs as well as API docs for the relevant software modules (Eg, FileReadingMessageSource), but I don't see any property or configuration option for turning off recursive descent into sub-directories.
What is the recommended practice here for scanning only files within the specified directory, but not recursing any deeper than that?
If you don't a recursion and scan the whole file tree, just don't use that watch service!
For the create and modify events you can configure a FileSystemPersistentAcceptOnceFileListFilter which checks for the file.lastModified(). I see you do that anyway, therefore it is not clear why do you need a watch service at all?
See some related discussed here: https://github.com/spring-projects/spring-integration/issues/3557.
If you still have some reasonable argument to use watch service for only a root dir, please add a comment into that issue and we will revise it respectively.
I tried to follow the spring integration SFTP adapter to setup the data feed from local directory to remote site with sftp outbound channel adapter. The feed basically works fine.
But I would like to use cached connection. With the official
guideline for cachingSessionFactory setup:
http://docs.spring.io/spring-integration/reference/html/sftp.html#sftp-session-caching
<bean id="sftpSessionFactory"
class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<property name="host" value="localhost"/>
</bean>
<bean id="cachingSessionFactory"
class="org.springframework.integration.file.remote.session.CachingSessionFactory">
<constructor-arg ref="sftpSessionFactory"/>
<constructor-arg value="10"/>
<property name="sessionWaitTimeout" value="1000"/>
</bean>
But seems even if I have the sftp-session-caching defined, the sftp session will be closed once a file is sent successfully?
Anyone has experience on this?
I defined ftp adapter to connect to ftp server, but I see ftp server log and don't see request sent to ftp server. my adapter code is :
<bean id="ftpClientFactory" class="org.springframework.integration.ftp.session.DefaultFtpSessionFactory">
<property name="host" value="127.0.0.1"/>
<property name="port" value="21"/>
<property name="username" value="banks_reader"/>
<property name="password" value="123456"/>
<property name="clientMode" value="2"/>
<property name="fileType" value="2"/>
</bean>
<int:channel id="inbound1">
<int:queue/>
</int:channel>
<int:channel id="outbound"/>
<int-ftp:outbound-gateway id="gateway1"
session-factory="ftpClientFactory"
request-channel="inbound1"
reply-channel="outbound"
reply-timeout="777"
auto-create-local-directory="false"
auto-startup="true"
filename-pattern="*"
remote-file-separator="/"
command="ls"
command-options="-1 -f"
expression="payload"
order="1"
mput-regex=".*">
<int:poller fixed-delay="1000"/>
</int-ftp:outbound-gateway>
which shows:
DEBUG PollingConsumer:208 - Received no Message during the poll, returning 'false' in application log
when change usename and password for test (fake pass and usename) Does not change anything and throw exception
Your config looks good, but you have missed an approach a bit.
<int-ftp:outbound-gateway> is an event-driven request/reply component and it can't do anything with FTP until there is no message in the inbound1.
Even if it is <queue>, the <poller> initiates its work only if there is no that Received no Message during the poll.
Since you use expression="payload", your requestMessage must contain a payload with remote dir from your FTP user home.
So, just send such a message to the inbound1 and let us know how it is!
UPDATE
To perform the LS command on the <int-ftp:outbound-gateway> periodically with the same dir as a payload you have to configure something like this:
<inbound-channel-adapter channel="inbound1" expresssion="'/'">
<poller fixed-delay="10000"/>
</inbound-channel-adapter>
Having that (and your <gateway>) case there is no need to have that inbound1 channel as a <queue>
I have, I suppose, really newbie question but the fact is I'm newbie in spring framework.
How can I upload files to, for example 'upload' folder loceted in root directory of my ftp server?
I have tried this:
My application context file:
<bean id="ftpClientFactory"
class="org.springframework.integration.ftp.session.DefaultFtpSessionFactory">
<property name="host" value="127.0.0.1"/>
<property name="username" value="test"/>
<property name="password" value="test"/>
<property name="clientMode" value="0"/>
<property name="fileType" value="2"/>
<property name="bufferSize" value="10000"/>
</bean>
<int:channel id="ftpChannel"/>
<int-ftp:outbound-channel-adapter id="outFtpAdapter"
channel="ftpChannel"
session-factory="ftpClientFactory"
remote-directory="/Users/test"/>
and my java code:
ConfigurableApplicationContext context =
new FileSystemXmlApplicationContext("/src/citrus/resources/citrus-context.xml");
MessageChannel ftpChannel = context.getBean("ftpChannel", MessageChannel.class);
File file = new File("/Users/test/test.txt");
Message<File> fileMessage = MessageBuilder.withPayload(file).build();
ftpChannel.send(fileMessage);
context.close();
But this example upload files to root directory.
Thanks in advance.
I've just tested it and work well:
<int-ftp:outbound-channel-adapter
id="sendFileToServer"
auto-create-directory="true"
session-factory="ftpSessionFactory"
remote-directory="/Users/test"/>
Where my FTP server is an embedded one and its root is:
[MY_USER_HOME]\Temp\junit7944189423444999123\FtpServerOutboundTests\
So the file is stored in the dir:
[MY_USER_HOME]\Temp\junit7944189423444999123\FtpServerOutboundTests\Users\test\
It is with the latest Spring Integration version.
Which is your version?
Using FTPClient upload MultipartFiles to apache FTP server
useful repo
Hope this help for future search.
I am using Spring Integration to download/upload files from FTP server.
How can I change remote-directory="/directory Name" dynamically in Spring FTP:Inbound-Channel.
My client will create a folder daily basically in "MM-dd-yy" format and copy all files there.
In "FTP:Inbound-channel" I did not find any way to configure this pattern. I basically have
to hardcord the directory or file names in configuration.
What I want is to set the path programatically. Because some times I need to download all files
from a direcotory or download only a specific file.
I found "remote-directory-expression="'directory'+'/'+ new java.text.SimpleDateFormat('dd-MM-yyyy').format(new java.util.Date())" can be set in FTP:Outbound-channel
is there any such attribute in FTP:InBound-channel
My configuration is like this:
<bean id="ftpClientFactory"
class="org.springframework.integration.ftp.session.DefaultFtpSessionFactory">
<property name="host" value="${host}" />
<property name="port" value="${availableServerPort}" />
<property name="username" value="${userid}" />
<property name="password" value="${password}" />
</bean>
<int-ftp:inbound-channel-adapter id="ftpInbound"
cache-sessions="false" channel="ftpChannel" session-factory="ftpClientFactory"
filename-pattern="*.txt" auto-create-local-directory="true"
delete-remote-files="false" remote-directory="/filedirectory"
local-directory="${local_directory}">
<int:poller fixed-rate="1000" />
</int-ftp:inbound-channel-adapter>
<int:channel id="ftpChannel">
<int:queue />
</int:channel>
I did not find a way to do all the above items.
Please let me know how can I achieve this.
You can't do it with the inbound adapter, but the <ftp:outbound-gateway/> can be used to achieve what you need; described here.
You can either use ls to list the files, followed a <splitter/> and another gateway using get; or you can use the mget command with a file name pattern in the expression.
The FTP sample has an example of using the gateway