Transfer files on different servers (sftp) using Spring Integration - spring

I have a requirement to build an application in SI which reads an input directory which may consist 1000s of file and copy them to remote servers, say 10 servers where a processor instance will pick them up for processing.The movement of file should be on round-robin fashion so that there is not additional burden on any server while processing them. To elaborate a little more - lets say we have 10 files in input directory then application should copy
file 1 on server1,
file2 on server2
.
.
.
....
file 10 on server 10.
Sequence doesn't matter what matters is that every server should have equal load.I am fairly new to Spring Integration but I found a sample to do sftp of file using SI
https://github.com/spring-projects/spring-integration-samples/tree/master/basic/sftp
but i am not sure how can I configure it for multiple servers and to have an algo to move files in round-robin fashion.
I will appreciate any tips or suggestion .
I am able to do sftp using below config.
<context:property-placeholder location="classpath:app.properties" />
<int-file:inbound-channel-adapter id="ReaderChannel"
directory="file:${input.file.dir}" filename-pattern="*.*"
prevent-duplicates="true" ignore-hidden="true" auto-startup="true">
<int:poller id="poller" fixed-rate="1" task-executor="myTaskExecutor" />
</int-file:inbound-channel-adapter>
<int-task:executor id="myTaskExecutor" pool-size="${file.concurrentFilesNum}" queue-capacity="0" rejection-policy="CALLER_RUNS" />
<int-sftp:outbound-channel-adapter id="sftpOutboundAdapter" session-factory="sftpSessionFactory" channel="ReaderChannel"
charset="UTF-8" remote-directory="${output.file.dir}" auto-startup="true">
<int-sftp:request-handler-advice-chain>
<int:retry-advice />
</int-sftp:request-handler-advice-chain>
</int-sftp:outbound-channel-adapter>
<beans:bean id="sftpSessionFactory" class="org.springframework.integration.file.remote.session.CachingSessionFactory">
<beans:constructor-arg ref="defaultSftpSessionFactory" />
</beans:bean>
<beans:bean id="defaultSftpSessionFactory"
class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<beans:property name="host" value="${sftp.host}" />
<beans:property name="privateKey" value="${sftp.private.keyfile}" />
<beans:property name="privateKeyPassphrase" value="${sftp.private.passphrase}" />
<beans:property name="port" value="${sftp.serverPort}" />
<beans:property name="user" value="${sftp.username}" />
<beans:property name="allowUnknownKeys" value="true" />
</beans:bean>

The round-robin is hidden in the DirectChannel with UnicastingDispatcher on the RoundRobinLoadBalancingStrategy.
So, when you have several subscriber to the same DirectChannel, the message will be dispatched to them in the round-robin.
What you need for your use-case is just configure 10 <int-sftp:outbound-channel-adapter> for each your remote server. And use the same simple <channel> definition for their channel attribute.
The <int-file:inbound-channel-adapter> should always send its message to that shared channel with default round-robin strategy.

Related

DefaultMessageListenerContainer recoveryInterval with specific retry count

We are using Spring Integration in our project and we have a requirement where If IBM MQ goes down then we will have to auto connect to IBM MQ when it is up. We have done this implementation using recoveryInterval option of org.springframework.jms.listener.DefaultMessageListenerContainer class. We have given recovery interval as 6 seconds so every 6 seconds system try to recover the MQ connection but now we have a requirement where we will have to do the autorecover twice only and after that if still MQ is down then stop the inbound adapter.
Is there any way in Spring Integration to mention the auto recovery retry count so that system will try to recover only for that retry count?
Below is my existing configuration.
<bean id="inQ" class="com.ibm.mq.jms.MQQueue">
<constructor-arg value="${mq.inbound.queue}" />
</bean>
<int:channel id="inbound" />
<int-jms:message-driven-channel-adapter
id="jmsIn" channel="inbound" container="messageListenerContainer"
acknowledge="transacted" auto-startup="false">
</int-jms:message-driven-channel-adapter>
<int:service-activator id="mainService"
input-channel="inbound" ref="messageListener" method="onMessage">
<int:request-handler-advice-chain>
<ref bean="retryWithBackoffAdviceSession" />
</int:request-handler-advice-chain>
</int:service-activator>
<bean id="messageListenerContainer"
class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="connectionFactory" ref="mqConnectionFactory" />
<property name="destination" ref="inQ" />
<property name="sessionTransacted" value="true" />
<property name="maxConcurrentConsumers" value="${maxConcurrentConsumers}" />
<property name="concurrentConsumers" value="${concurrentConsumers}" />
<property name="receiveTimeout" value="${receiveTimeout}" />
<property name="recoveryInterval" value="60000" />
<property name="autoStartup" value="${autoStartup}" />
</bean>
Thanks
Sach
As an alternative to the recoveryInterval, you can now specify a Backoff instead (see the docs).
It doesn't provide a mechanism to stop the container but an appropriate backoff can effectively do what you want.
You would then need to programmatically stop/start to kick it off again.

SFTP Spring Integrations rm command

I am wondering if someone can assist, I know I am doing this wrong and I'm tearing my hair out. My goal is to delete any files with a .txt extension in a remote directory using Spring Integrations SFTP in a Spring Batch job. It is my understanding that I do not have to ls remote files to remove them and can just issue an rm command on *.txt for a given directory however I may be incorrect?
I have the following SFTP configuration
<bean id="sftpSessionFactory"
class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<property name="host" value="${host}" />
<property name="port" value="${port}" />
<property name="user" value="${user}" />
<property name="privateKey" value="file:${privateKey}" />
<property name="password" value="${password}" />
</bean>
<int:channel id="rmChannel"/>
<int-sftp:outbound-gateway
session-factory="sftpSessionFactory"
request-channel="rmChannel"
remote-file-separator="/"
command="rm"
expression="headers['file_remoteDirectory'] + headers['file_remoteFile']" />
<bean id="cleanRemoteDirectoryTasklet" class="com.example.batch.job.integration.SFTPRmTasklet" scope="step">
<property name="channel" ref="rmChannel" />
<property name="filePatternToDelete" value="*.txt" />
<property name="targetDirectory" value="${remoteDirectory}"/> // edit removed 'file:' notation
</bean>
I believe I am OK to this point an my problem is executing this flow in the Java implementation in SFTPRmTasklet, I'm not sure how to construct the message to initiate the sftp remove. Currently I have something like this simply to kick it off I know that my payload is wrong.
Message<String> rmRequest = MessageBuilder.withPayload("/my/target/dir")
.setHeader("file_remoteDirectory", targetDirectory)
.setHeader("file_remoteFile", filePatternToDelete)
.build();
channel.send(rmRequest)
ultimately this yields an exception
org.springframework.integration.MessagingException: org.springframework.core.NestedIOException: Failed to remove file: 2: No such file
UPDATE 1
So I decided to target just one remote file and changed filePatternToDelete to test.txt. After a bit of debugging I realised that AbstractRemoteFileOutBoundGateway was evaluating my remoteFilePath to /my/target/dirtest.txt and remote filename to dirtest.txt, which is obviously not what I wanted so I added a trailing to / to the target directory in my properties file and this sorted out this error great!
I can now delete the file from the remote server as I wished to do however I received an error around no reply-channel so I have added the following channel
<int:channel id="end"/>
and modified my outbound gateway
<int-sftp:outbound-gateway
session-factory="sftpSessionFactory"
request-channel="rmChannel"
reply-channel="end"
remote-file-separator="/"
command="rm"
expression="headers['file_remoteDirectory'] + headers['file_remoteFile']" />
and now get an error around no subscribers for this channel. Progress at least and in case you hadn't guessed I'm pretty new to Spring!
If you are not interested in the result of the rm, set the reply-channel to nullChannel.
It's like /dev/null on unix.

how to trigger many ftp inbound-channel-adapter from one poller to connect ftp server?

First, thank you for your attention
i defined two ftp inbound-channel-adapter in my spring integration project,the adapters with diferent configuration but have share session factory to connect ftp server, each adapters have one poller to trigger,i want to see the bellow code is correct?
is efficiency?
<bean id="ftpClientFactory"
class="org.springframework.integration.ftp.session.DefaultFtpSessionFactory">
<property name="host" value="127.0.0.1"/>
<property name="port" value="21"/>
<property name="username" value="banks_reader"/>
<property name="password" value="123456"/>
</bean>
<bean id="myFilter" class="com.ali.util.FtpFilter"/>
<bean id="onceFilter" class="com.ali.util.OnceFilter"/>
<int-ftp:inbound-channel-adapter id="ftpInbound1"
channel="inboundFileChannel"
**session-factory="ftpClientFactory"**
charset="UTF-8"
auto-create-local-directory="true"
delete-remote-files="true"
remote-directory="/directoryA"
remote-file-separator="/"
temporary-file-suffix=".writing"
local-filter="myFilter"
filter="onceFilter"
local-directory="file:output">
**<int:poller fixed-rate="5000"/>**
</int-ftp:inbound-channel-adapter>
<int-ftp:inbound-channel-adapter id="ftpInbound2"
channel="inboundFileChannel"
**session-factory="ftpClientFactory"**
charset="UTF-8"
auto-create-local-directory="true"
delete-remote-files="true"
remote-directory="/directoryB"
remote-file-separator="/"
temporary-file-suffix=".writing"
local-filter="myFilter"
filter="onceFilter"
local-directory="file:output">
**<int:poller fixed-rate="5000"/>**
</int-ftp:inbound-channel-adapter>
Is there another way?
That is the correct technique using channel adapters.
Or, you could use an outbound gateway instead, and fetch (ls + get or mget) files from both directories in turn.

Could not fetch messages from Webpshere MQ sometimes even though the application and websphere mq is up and running

I have encountered a problem in fetching the messages from Websphere MQ.
We have a application running in spring TC server.
Application is using spring integrator JMS adaptor to receive the messages from Websphere MQ server.
Following is the spring configuration we used.
Problem we are facing is, sometime even though the websphere mq is up and running and application is up and running, the application is not able to fetch the messages so that messages are falling to websphere mq dead letter queue
Is there any possibility for the above scenario? We are not able to reproduce this scenario
<beans:bean id="connectionFactory" class="com.ibm.mq.jms.MQQueueConnectionFactory">
<beans:property name="transportType">
<util:constant static-field="com.ibm.mq.jms.JMSC.MQJMS_TP_CLIENT_MQ_TCPIP" />
</beans:property>
<beans:property name="queueManager" value="queueManager" />
<beans:property name="hostName" value="hostName" />
<beans:property name="channel" value="channel" />
<beans:property name="port" value="port" />
<beans:property name="clientReconnectOptions">
<util:constant static-field="com.ibm.msg.client.wmq.WMQConstants.WMQ_CLIENT_RECONNECT"/>
</beans:property>
</beans:bean>
<beans:bean id="mqSeriesConnectionFactory" class="org.springframework.jms.connection.UserCredentialsConnectionFactoryAdapter">
<beans:property name="username" value="username"/>
<beans:property name="password" value="password"/>
<beans:property name="targetConnectionFactory" ref="connectionFactory"/>
</beans:bean>
<beans:bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
<beans:property name="connectionFactory" ref="mqSeriesConnectionFactory" />
</beans:bean>
<beans:bean id="mqQueueG3Receive" class="com.ibm.mq.jms.MQQueue">
<beans:constructor-arg value="receivequeue" />
<beans:property name="targetClient" value="1"/>
</beans:bean>
<beans:bean id="mqQueueG3Send" class="com.ibm.mq.jms.MQQueue">
<beans:constructor-arg value="sendqueue" />
<beans:property name="targetClient" value="1"/>
</beans:bean>
<jms:message-driven-channel-adapter id="MessageManagerJmsAdapter"
connection-factory="mqSeriesConnectionFactory"
destination="mqQueueG3Send"
concurrent-consumers="7"
max-concurrent-consumers="40"
transaction-manager="jmsTransactionManager"
message-converter="resultMessageConverter"
channel="MessageManagerIncomingChannel" />
<channel id="g3MessageManagerIncomingChannel" />
<service-activator id="MessageManagerActivator"
input-channel="MessageManagerIncomingChannel"
ref="MessageManager"
method="manageMessage" />
Thanks
WebSphere MQ is a little finicky when it comes to Spring Integration JMS. instead of building up the message-driven-channel-adapter with connection settings, you may want to try configuring your own DefaultMessageListenerContainer and using that. things to look out for include;
sessionTransacted = true
transactionManager = [ref]
so this would look like;
<int-jms:message-driven-channel-adapter channel="myChannel" container="myContainer"/>
<bean id="myContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="connectionFactory" ref="myConnectionFactory"/>
<property name="destination" ref="myQueue"/>
<property name="sessionTransacted" value="true"/>
<property name="transactionManager" ref="myTransactionManager"/>
</bean>
If they are going to the DLQ it means delivery failed, not that the app could not "fetch the message". I would expect there to be an error logged.
Another thing to look for is if the container's threads (7) are blocked somewhere in your MessageManager bean. Take a thread dump (jstack or visualVM) to find out what the container threads are doing.
Finally, trace logging will show the container thread activity when idle.

Downloading files from FTP Server where directory changed dynamically

I am using Spring Integration to download/upload files from FTP server.
How can I change remote-directory="/directory Name" dynamically in Spring FTP:Inbound-Channel.
My client will create a folder daily basically in "MM-dd-yy" format and copy all files there.
In "FTP:Inbound-channel" I did not find any way to configure this pattern. I basically have
to hardcord the directory or file names in configuration.
What I want is to set the path programatically. Because some times I need to download all files
from a direcotory or download only a specific file.
I found "remote-directory-expression="'directory'+'/'+ new java.text.SimpleDateFormat('dd-MM-yyyy').format(new java.util.Date())" can be set in FTP:Outbound-channel
is there any such attribute in FTP:InBound-channel
My configuration is like this:
<bean id="ftpClientFactory"
class="org.springframework.integration.ftp.session.DefaultFtpSessionFactory">
<property name="host" value="${host}" />
<property name="port" value="${availableServerPort}" />
<property name="username" value="${userid}" />
<property name="password" value="${password}" />
</bean>
<int-ftp:inbound-channel-adapter id="ftpInbound"
cache-sessions="false" channel="ftpChannel" session-factory="ftpClientFactory"
filename-pattern="*.txt" auto-create-local-directory="true"
delete-remote-files="false" remote-directory="/filedirectory"
local-directory="${local_directory}">
<int:poller fixed-rate="1000" />
</int-ftp:inbound-channel-adapter>
<int:channel id="ftpChannel">
<int:queue />
</int:channel>
I did not find a way to do all the above items.
Please let me know how can I achieve this.
You can't do it with the inbound adapter, but the <ftp:outbound-gateway/> can be used to achieve what you need; described here.
You can either use ls to list the files, followed a <splitter/> and another gateway using get; or you can use the mget command with a file name pattern in the expression.
The FTP sample has an example of using the gateway

Resources