Polling directory with multiple sub-directories - spring

I am trying to build a simple utility that will copy the files from multiple directories from one sftp server to another server.
I tried use sftp outbound gateway to poll single high level directory with command "mget" , but it did not work. So I thought of writing two inbound adapters ( not a good solution, but still wanted this to be done badly !) .
<int-sftp:inbound-channel-adapter
id="pdbInbound"
session-factory="sftpSessionFactory"
auto-create-local-directory="true" delete-remote-files="true"
filename-pattern="*.*" remote-directory="${remote.pdb.directory}"
local-directory="${local.pdb.directory}">
<int:poller fixed-rate="5000"/>
</int-sftp:inbound-channel-adapter>
<int-sftp:inbound-channel-adapter
id="galaxyInbound"
session-factory="sftpSessionFactory"
auto-create-local-directory="true" delete-remote-files="true"
filename-pattern="*.*" remote-directory="${remote.galaxy.directory}"
local-directory="${local.galaxy.directory}" >
<int:poller fixed-rate="5000"/>
</int-sftp:inbound-channel-adapter>
Above code works perfectly fine and files are copied to local directories as expected.
Problem appears when I need to transfer these files to remote directory with the same directory structure as that of source directory. I could not achieve it using sftp-outbound gateway with command = "mput" and command-options= "-R". So, I tried to write two outbound adapters as below. But only one directory is written to remote.
Any idea what is going wrong here ?
<int:service-activator input-channel="pdbInbound" output-channel="pdbOutbound" expression="payload"/>
<int:service-activator input-channel="galaxyInbound" output-channel="galaxyOutbound" expression="payload"/>
<int-sftp:outbound-channel-adapter id="sftPdbOutboundAdapter" auto-create-directory="true"
session-factory="sftpSessionFactory"
auto-startup="true"
channel="pdbOutbound"
charset="UTF-8"
remote-file-separator="/"
remote-directory="${remote.out.pdb.directory}"
mode="REPLACE">
</int-sftp:outbound-channel-adapter>
<int-sftp:outbound-channel-adapter id="sftpGalaxyOutboundAdapter" auto-create-directory="true"
auto-startup="true"
session-factory="sftpSessionFactory"
channel="galaxyOutbound"
charset="UTF-8"
remote-file-separator="/"
remote-directory="${remote.out.galaxy.directory}"
mode="REPLACE">
</int-sftp:outbound-channel-adapter>
<int:poller default="true" fixed-delay="50"/>
Note: I am using same sftp server (but different directories) for inbound and outbound files for testing purpose.

You need to explain your issues in more details - "did not work" is woefully inadequate and you won't get much help here with such a question. You need to show what you tried and what you observed.
There are test cases for both recursive mget and recursive mput.
The directory structure for the tests is shown in a comment at the top of that file.
I suggest you compare those with what you tried and come back here if you have a specific question/observation. Best thing to do to solve these issues is to turn on DEBUG logging; including for jsch.

Related

FTP Spring Integration with Inbound Streaming Channel Adapter

I am using spring-integration with ftp. When i use int-ftp:inbound-channel-adapter, then it is working fine, but int-ftp:inbound-streaming-channel-adapter yields the following error:
cvc-complex-type.2.4.c: The matching wildcard is strict, but no declaration can be found for element 'int-ftp:inbound-streaming-channel-adapter
What could I be missing?
The relevant portion of my code is
<int-ftp:inbound-streaming-channel-adapter id="ftpInbound"
channel="ftpChannel"
session-factory="ftpClientFactory"
filename-pattern="*.txt"
filename-regex=".*\.txt"
filter="customFilter"
remote-file-separator="/"
comparator="comparator"
remote-directory-expression="'/OUT/SDI402_CARATT_JD'">
<int:poller fixed-rate="1000" />
</int-ftp:inbound-streaming-channel-adapter>
Use in your dependencies spring integration 4.3.1.RELEASE. That solved the exact same problem in my case.

How to integrate legacy executables into Spring Integration application?

I have some legacy programs in C that work based on the input file and the result go to the output file. Both files are specified in the program arguments. So the call looks like the following:
prj.exe a.dat a.out
Based on Artem Bilan suggestion I created the project with the following Spring Configuration file. It works in terms of invoking executable! However, I still have the problem with the outbound channel. First, it contains nothing and I am getting the error "unsupported Message payload type". Second, what is more important, I need to process the output file a.out by a Java program. What is the best way to organize this workflow? Is it possible to substitute the useless in this case inbound-channel-adapter to something useful?
<int-file:inbound-channel-adapter id="producer-file-adapter"
channel="inboundChannel" directory="file:/Users/anarinsky/springint/chem"
prevent-duplicates="true">
<int:poller fixed-rate="5000" />
</int-file:inbound-channel-adapter>
<int:channel id="inboundChannel" />
<int:channel id="outboundChannel" />
<int:service-activator input-channel="inboundChannel" output-channel="outboundChannel"
expression="new ProcessBuilder('/Users/anarinsky/springint/chem/prj', '/Users/anarinsky/springint/chem/a.dat', '/Users/anarinsky/springint/chem/a.out').start()">
</int:service-activator>
<int-file:outbound-channel-adapter
channel="outboundChannel" id="consumer-file-adapter"
directory="file:/Users/anarinsky/springint/chem"/>
Something like this:
<int:service-activator expression="new ProcessBuilder('prj.exe', 'a.dat', 'a.out').start()"/>
?

SFTP Spring Integrations rm command

I am wondering if someone can assist, I know I am doing this wrong and I'm tearing my hair out. My goal is to delete any files with a .txt extension in a remote directory using Spring Integrations SFTP in a Spring Batch job. It is my understanding that I do not have to ls remote files to remove them and can just issue an rm command on *.txt for a given directory however I may be incorrect?
I have the following SFTP configuration
<bean id="sftpSessionFactory"
class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<property name="host" value="${host}" />
<property name="port" value="${port}" />
<property name="user" value="${user}" />
<property name="privateKey" value="file:${privateKey}" />
<property name="password" value="${password}" />
</bean>
<int:channel id="rmChannel"/>
<int-sftp:outbound-gateway
session-factory="sftpSessionFactory"
request-channel="rmChannel"
remote-file-separator="/"
command="rm"
expression="headers['file_remoteDirectory'] + headers['file_remoteFile']" />
<bean id="cleanRemoteDirectoryTasklet" class="com.example.batch.job.integration.SFTPRmTasklet" scope="step">
<property name="channel" ref="rmChannel" />
<property name="filePatternToDelete" value="*.txt" />
<property name="targetDirectory" value="${remoteDirectory}"/> // edit removed 'file:' notation
</bean>
I believe I am OK to this point an my problem is executing this flow in the Java implementation in SFTPRmTasklet, I'm not sure how to construct the message to initiate the sftp remove. Currently I have something like this simply to kick it off I know that my payload is wrong.
Message<String> rmRequest = MessageBuilder.withPayload("/my/target/dir")
.setHeader("file_remoteDirectory", targetDirectory)
.setHeader("file_remoteFile", filePatternToDelete)
.build();
channel.send(rmRequest)
ultimately this yields an exception
org.springframework.integration.MessagingException: org.springframework.core.NestedIOException: Failed to remove file: 2: No such file
UPDATE 1
So I decided to target just one remote file and changed filePatternToDelete to test.txt. After a bit of debugging I realised that AbstractRemoteFileOutBoundGateway was evaluating my remoteFilePath to /my/target/dirtest.txt and remote filename to dirtest.txt, which is obviously not what I wanted so I added a trailing to / to the target directory in my properties file and this sorted out this error great!
I can now delete the file from the remote server as I wished to do however I received an error around no reply-channel so I have added the following channel
<int:channel id="end"/>
and modified my outbound gateway
<int-sftp:outbound-gateway
session-factory="sftpSessionFactory"
request-channel="rmChannel"
reply-channel="end"
remote-file-separator="/"
command="rm"
expression="headers['file_remoteDirectory'] + headers['file_remoteFile']" />
and now get an error around no subscribers for this channel. Progress at least and in case you hadn't guessed I'm pretty new to Spring!
If you are not interested in the result of the rm, set the reply-channel to nullChannel.
It's like /dev/null on unix.

Downloading files from FTP Server where directory changed dynamically

I am using Spring Integration to download/upload files from FTP server.
How can I change remote-directory="/directory Name" dynamically in Spring FTP:Inbound-Channel.
My client will create a folder daily basically in "MM-dd-yy" format and copy all files there.
In "FTP:Inbound-channel" I did not find any way to configure this pattern. I basically have
to hardcord the directory or file names in configuration.
What I want is to set the path programatically. Because some times I need to download all files
from a direcotory or download only a specific file.
I found "remote-directory-expression="'directory'+'/'+ new java.text.SimpleDateFormat('dd-MM-yyyy').format(new java.util.Date())" can be set in FTP:Outbound-channel
is there any such attribute in FTP:InBound-channel
My configuration is like this:
<bean id="ftpClientFactory"
class="org.springframework.integration.ftp.session.DefaultFtpSessionFactory">
<property name="host" value="${host}" />
<property name="port" value="${availableServerPort}" />
<property name="username" value="${userid}" />
<property name="password" value="${password}" />
</bean>
<int-ftp:inbound-channel-adapter id="ftpInbound"
cache-sessions="false" channel="ftpChannel" session-factory="ftpClientFactory"
filename-pattern="*.txt" auto-create-local-directory="true"
delete-remote-files="false" remote-directory="/filedirectory"
local-directory="${local_directory}">
<int:poller fixed-rate="1000" />
</int-ftp:inbound-channel-adapter>
<int:channel id="ftpChannel">
<int:queue />
</int:channel>
I did not find a way to do all the above items.
Please let me know how can I achieve this.
You can't do it with the inbound adapter, but the <ftp:outbound-gateway/> can be used to achieve what you need; described here.
You can either use ls to list the files, followed a <splitter/> and another gateway using get; or you can use the mget command with a file name pattern in the expression.
The FTP sample has an example of using the gateway

H2 database console, how the setting -webAllowOthers works?

I'm starting H2 console from spring:
<spring:bean id="H2WebServer" class="org.h2.tools.Server"
scope="singleton" factory-method="createWebServer" init-method="start"
destroy-method="stop">
<spring:constructor-arg value="-web,-webAllowOthers,true,-webPort,8082" />
</spring:bean>
H2 version is 1.3.160
I do not want the console to be accesible from other computers in my network.
The documentation says to use setting webAllowOthers to allow/or not to other computers.
But if i set "-webAllowOthers,false", the console is still available in my local network.
I also checked .h2.server.properties file.
How should the setting work?
If you don't want to allow other computers, then remove the -webAllowOthers:
Like:
<spring:constructor-arg value="-web,-webPort,8082" />
else keep just the -webAllowOthers
<spring:constructor-arg value="-web,-webAllowOthers,-webPort,8082" />
If setting removed,
then it respondes with "Sorry, remote connections are disabled on this server".
You can always use firewall rules ;) iptables on linux and such.

Resources