https://medium.com/nerd-for-tech/retrieving-files-from-ftp-server-using-spring-integration-5ccc4a972eaf
I'm implementing FTP behavior based on the above site.
In the above site, I've only added passive mode to the ftp settings.
#Bean
public DefaultFtpSessionFactory sf() {
DefaultFtpSessionFactory sf = new DefaultFtpSessionFactory();
sf.setHost(host);
sf.setPort(port);
sf.setUsername(username);
sf.setPassword(password);
sf.setClientMode(FTPClient.PASSIVE_LOCAL_DATA_CONNECTION_MODE);
return sf;
}
And I want to check the passive port when communicating in practice.
#ServiceActivator(inputChannel = "ftpMGET")
#Bean
public FtpOutboundGateway getFiles() {
DefaultFtpSessionFactory ftpSessionFactory = sf();
FtpOutboundGateway gateway = new FtpOutboundGateway(ftpSessionFactory, "mget", "payload");
gateway.setAutoCreateDirectory(true);
gateway.setLocalDirectory(new File(downloadPath));
gateway.setFileExistsMode(FileExistsMode.APPEND);
gateway.setFilter(new AcceptOnceFileListFilter<>());
gateway.setOutputChannelName("fileResults");
gateway.setOption(AbstractRemoteFileOutboundGateway.Option.RECURSIVE);
System.out.println(ftpSessionFactory.getSession().getClientInstance().getPassivePort());
return gateway;
}
At the end, I added a passivePort with System.out.println.
It keeps saying -1, but how do I check the passive port when downloading a file from a real server?!
The FTPClient.getPassivePort cannot return any actual value, before you transfer anything. Each transfer uses a different (passive) port. Until you actually transfer anything, no passive port is involved.
As the documentation of the FTPClient.getPassivePort method says:
This method only returns a valid value AFTER a data connection has been opened after a call to enterLocalPassiveMode(). This is because FTPClient sends a PASV command to the server only just before opening a data connection, and not when you call enterLocalPassiveMode().
Related
I have created an application that adds inbound adaptors at run time for ftp servers and registers them for removing at a certain stage if needed, this app will pull a csv file from ftp server(s) and place it in my local in a folder having the name of the ftp server, so every server I add will have a separate local folder created and the csv file is saved in it, now this is accomplished smoothly, the second part is I want to change the format of that file and then send it back to the respective server, so basically I need to use outbound adaptor, in this case I would need to create outbound adaptors at run time at the same time when creating inbound adaptor or adding a server, this should be done through controller same as the inbound, I searched for possible solutions and tried a one that is below but did not work or did not perform any sending of files to destination, any solution on how I can accomplish this?
In the configuration class I added the below:
public IntegrationFlow ftpOutboundFlow(Branch myBranch){
return IntegrationFlows.from(OUTBOUND_CHANNEL)
.handle(Ftp.outboundAdapter(createNewFtpSessionFactory(myBranch), FileExistsMode.FAIL)
.useTemporaryFileName(true)
.remoteFileSeparator("/")
//.fileNameExpression("BEY/FEFOexportBEY.csv")
.remoteDirectory(myBranch.getFolderPath()))
.get();
}
#MessagingGateway
public interface MyGateway {
#Gateway(requestChannel = OUTBOUND_CHANNEL)
void sendToFtp(File file);
}
#Bean
public IntegrationFlow ftpOutboundChannel() {
return IntegrationFlows.from(OUTBOUND_CHANNEL)
.publishSubscribeChannel(p -> p
.subscribe(t -> t.handle(System.out::println)))
/*.transform(p -> {
LOG.info("Outbound intermediate Channel, message=rename file:" + p);
return p;
})*/
.channel(new NullChannel())
.get();
}
And in the Controller class
#RequestMapping("/branch/showbranch/{id}")
public String getBranch (#PathVariable String id, Model model){
model.addAttribute("branch", branchService.getById(Long.valueOf(id)));
addFlowftp(id);
addFlowftpOutbound(id);
return "/branch/showbranch";
}
private void addFlowFtp(String name) {
branch = branchService.getById(Long.valueOf(name));
System.out.println(branch.getBranchCode());
IntegrationFlow flow = ftIntegration.fileInboundFlowFromFTPServer(branch);
this.flowContext.registration(flow).id(name).register();
}
private void addFlowftpOutbound(String name) {
branch = branchService.getById(Long.valueOf(name));
System.out.println(branch.getBranchCode());
IntegrationFlow flow = ftIntegration.ftpOutboundFlow(branch);
// this.flowContext.registration(flow).id(name).register();
myGateway.sendToFtp(new File("BEY/FEFOexportBEY.csv"));
}
Here is what I get in the consol as error when I enable the register before sending the file:
java.lang.IllegalArgumentException: An IntegrationFlow 'IntegrationFlowRegistration{integrationFlow=StandardIntegrationFlow{integrationComponents={org.springframework.integration.ftp.inbound.FtpInboundFileSynchronizer#aadab28=98.org.springframework.integration.ftp.inbound.FtpInboundFileSynchronizer#0, org.springframework.integration.config.SourcePollingChannelAdapterFactoryBean#aa290b3=stockInboundPoller, org.springframework.integration.transformer.MethodInvokingTransformer#e5f85cd=98.org.springframework.integration.transformer.MethodInvokingTransformer#0, 98.channel#0=98.channel#0, org.springframework.integration.config.ConsumerEndpointFactoryBean#319dff2=98.org.springframework.integration.config.ConsumerEndpointFactoryBean#0}}, id='98', inputChannel=null}' with flowId '98' is already registered.
An existing IntegrationFlowRegistration must be destroyed before overriding.
After my second trial where I removed the registration from the first method and only tried with the second method but nothing was sent to the FTP:
GenericMessage [payload=BEY\FEFOexportBEY.csv, headers={id=43cfc2db-41e9-0866-8e4c-8e95968189ff, timestamp=1542702869007}]
2018-11-20 10:34:29.011 INFO 13716 --- [nio-8081-exec-6] f.s.s.configuration.FTIntegration : Outbound intermediate Channel, message=rename file:BEY\FEFOexportBEY.csv
You need to perform this.flowContext.registration(flow).id(name).register(); before sending the file via myGateway.sendToFtp(new File("BEY/FEFOexportBEY.csv"));.
That's first.
Other concern I see in your code that you have a ftpOutboundChannel bean for an IntegrationFlow which is subscribed to the same OUTBOUND_CHANNEL. If that one is not declared as a PublishSubscribeChannel, then you'll end up with the round-robin distribution. And I believe you would like to have file sent to the FTP and logged. So, indeed you need to declare that channel as a PublishSubscribeChannel.
You don't have any error because that OUTBOUND_CHANNEL has your ftpOutboundChannel IntegrationFlow as subscriber.
I use spring integration sftp to download and upload files.In the document ,I found
Spring Integration supports sending and receiving files over SFTP by providing three client side endpoints: Inbound Channel Adapter, Outbound Channel Adapter, and Outbound Gateway
When I want to download files I must assign the local directory and when I want to upload files I must assign the remote directory.But if I can't assign the directory when I write the code such as my directory is association with date.How can I assign the directory at runtime?
Here is my code:
#Bean
public SessionFactory<LsEntry> sftpSessionFactory(){
DefaultSftpSessionFactory defaultSftpSessionFactory = new DefaultSftpSessionFactory();
defaultSftpSessionFactory.setHost(host);
defaultSftpSessionFactory.setPort(Integer.parseInt(port));
defaultSftpSessionFactory.setUser(username);
defaultSftpSessionFactory.setPassword(password);
defaultSftpSessionFactory.setAllowUnknownKeys(true);
return new CachingSessionFactory<LsEntry>(defaultSftpSessionFactory);
}
#Bean
public SftpRemoteFileTemplate sftpRemoteFileTemplate(){
SftpRemoteFileTemplate sftpRemoteFileTemplate = new SftpRemoteFileTemplate(sftpSessionFactory());
return sftpRemoteFileTemplate;
}
#Bean
#ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler handlerGet() {
SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(sftpSessionFactory(), "mget", "payload");
sftpOutboundGateway.setLocalDirectory(new File(localDirectory));
sftpOutboundGateway.setFilter(new SftpSimplePatternFileListFilter("*.txt"));
sftpOutboundGateway.setSendTimeout(1000);
return sftpOutboundGateway;
}
In the messageHandler,I must assign the localDirectory in the outboundGateway. And when I want change my localDirectory by days.I must download the file to the localDirectory and move to the target directory. How can I assign the localDirectory at runtime .such as today I download to 20170606/ and tomorrow I download to 20170607 ?
edit
this is my option and test
public interface OutboundGatewayOption {
#Gateway(requestChannel = "sftpChannel")
public List<File> getFiles(String dir);
}
#Test
public void test2(){
outboundGatewayOption.getFiles("upload/20160920/");
}
sftpOutboundGateway.setLocalDirectoryExpression(
new SpelExpressionParser().parseExpression("headers['whereToPutTheFiles']");
or parseExpression("#someBean.getDirectoryName(payload)")
etc.
The expression must evaluate to a String representing the directory absolute path.
While evaluating the expression, the remote directory is available as a variable #remoteDirectory.
I'm new to Spring Framework and, indeed, I'm learning and using Spring Boot. Recently, in the app I'm developing, I made Quartz Scheduler work, and now I want to make Spring Integration work there: FTP connection to a server to write and read files from.
What I want is really simple (as I've been able to do so in a previous java application). I've got two Quartz Jobs scheduled to fired in different times daily: one of them reads a file from a FTP server and another one writes a file to a FTP server.
I'll detail what I've developed so far.
#SpringBootApplication
#ImportResource("classpath:ws-config.xml")
#EnableIntegration
#EnableScheduling
public class MyApp extends SpringBootServletInitializer {
#Autowired
private Configuration configuration;
//...
#Bean
public DefaultFtpsSessionFactory myFtpsSessionFactory(){
DefaultFtpsSessionFactory sess = new DefaultFtpsSessionFactory();
Ftp ftp = configuration.getFtp();
sess.setHost(ftp.getServer());
sess.setPort(ftp.getPort());
sess.setUsername(ftp.getUsername());
sess.setPassword(ftp.getPassword());
return sess;
}
}
The following class I've named it as a FtpGateway, as follows:
#Component
public class FtpGateway {
#Autowired
private DefaultFtpsSessionFactory sess;
public void sendFile(){
// todo
}
public void readFile(){
// todo
}
}
I'm reading this documentation to learn to do so. Spring Integration's FTP seems to be event driven, so I don't know how can I execute either of the sendFile() and readFile() from by Jobs when the trigger is fired at an exact time.
The documentation tells me something about using Inbound Channel Adapter (to read files from a FTP?), Outbound Channel Adapter (to write files to a FTP?) and Outbound Gateway (to do what?):
Spring Integration supports sending and receiving files over FTP/FTPS by providing three client side endpoints: Inbound Channel Adapter, Outbound Channel Adapter, and Outbound Gateway. It also provides convenient namespace-based configuration options for defining these client components.
So, I haven't got it clear as how to follow.
Please, could anybody give me a hint?
Thank you!
EDIT:
Thank you #M. Deinum. First, I'll try a simple task: read a file from the FTP, the poller will run every 5 seconds. This is what I've added:
#Bean
public FtpInboundFileSynchronizer ftpInboundFileSynchronizer() {
FtpInboundFileSynchronizer fileSynchronizer = new FtpInboundFileSynchronizer(myFtpsSessionFactory());
fileSynchronizer.setDeleteRemoteFiles(false);
fileSynchronizer.setPreserveTimestamp(true);
fileSynchronizer.setRemoteDirectory("/Entrada");
fileSynchronizer.setFilter(new FtpSimplePatternFileListFilter("*.csv"));
return fileSynchronizer;
}
#Bean
#InboundChannelAdapter(channel = "ftpChannel", poller = #Poller(fixedDelay = "5000"))
public MessageSource<File> ftpMessageSource() {
FtpInboundFileSynchronizingMessageSource source = new FtpInboundFileSynchronizingMessageSource(inbound);
source.setLocalDirectory(new File(configuracion.getDirFicherosDescargados()));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new AcceptOnceFileListFilter<File>());
return source;
}
#Bean
#ServiceActivator(inputChannel = "ftpChannel")
public MessageHandler handler() {
return new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
Object payload = message.getPayload();
if(payload instanceof File){
File f = (File) payload;
System.out.println(f.getName());
}else{
System.out.println(message.getPayload());
}
}
};
}
Then, when the app is running, I put a new csv file intro "Entrada" remote folder, but the handler() method isn't run after 5 seconds... I'm doing something wrong?
Please add #Scheduled(fixedDelay = 5000) over your poller method.
You should use SPRING BATCH with tasklet. It is far easier to configure bean, crone time, input source with existing interfaces provided by Spring.
https://www.baeldung.com/introduction-to-spring-batch
Above example is annotation and xml based both, you can use either.
Other benefit Take use of listeners and parallel steps. This framework can be used in Reader - Processor - Writer manner as well.
We have a use case to download files from FTP, and there is a strange behavior that the ftp inbound adapter stops to work after network resets, here is the steps to reproduce the problem:
start application
application starts to download files from ftp server to local
there are filename.writing file appearing in defined local directory
pull out the network cable (to simulate a network reset situation)
application stops to download file (obviously no network connection)
plug in the network cable.
download is not restarted or reset, application stays still..
there is no LOG at all to identify this problem.
Thanks in advance!
UPDATE
This problem should be fixed by adding timeout defSession.setConnectTimeout(Integer.valueOf(env.getProperty("ftp.timeout.connect")));
AND The code below is a WORKING EXAMPLE on FTP reading client.
Here are the code snippet:
#Bean
public DefaultFtpSessionFactory ftpSessionFactory() {
DefaultFtpSessionFactory defSession = new DefaultFtpSessionFactory();
defSession.setUsername(env.getProperty("ftp.username"));
defSession.setPassword(env.getProperty("ftp.password"));
defSession.setPort(21);
defSession.setHost(env.getProperty("ftp.host"));
defSession.setClientMode(FTPClient.PASSIVE_LOCAL_DATA_CONNECTION_MODE);
defSession.setControlEncoding("UTF-8");
return defSession;
}
#Bean
PollableChannel ftpChannel() {
return new QueueChannel(Integer.valueOf(env.getProperty("ftp.channel.size")));
}
#Bean
public FtpInboundFileSynchronizer ftpInboundFileSynchronizer() {
FtpInboundFileSynchronizer ftpInboundFileSynchronizer = new FtpInboundFileSynchronizer(ftpSessionFactory());
ftpInboundFileSynchronizer.setDeleteRemoteFiles(Boolean.valueOf(env.getProperty("ftp.directory.delete")));
FtpRegexPatternFileListFilter ftpRegexPatternFileListFilter = new FtpRegexPatternFileListFilter(pattern);
ftpInboundFileSynchronizer.setFilter(ftpRegexPatternFileListFilter);
ftpInboundFileSynchronizer.setRemoteDirectory(env.getProperty("ftp.directory.remote"));
return ftpInboundFileSynchronizer;
}
#Bean
#InboundChannelAdapter(value = "ftpChannel")
public FtpInboundFileSynchronizingMessageSource ftpInboundFileSynchronizingMessageSource() {
FtpInboundFileSynchronizingMessageSource ftpInboundFileSynchronizingMessageSource = new FtpInboundFileSynchronizingMessageSource(ftpInboundFileSynchronizer());
ftpInboundFileSynchronizingMessageSource.setLoggingEnabled(true);
ftpInboundFileSynchronizingMessageSource.setCountsEnabled(true);
ftpInboundFileSynchronizingMessageSource.setAutoCreateLocalDirectory(true);
ftpInboundFileSynchronizingMessageSource.setLocalDirectory(new File(env.getProperty("ftp.directory.local")));
return ftpInboundFileSynchronizingMessageSource;
}
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata defaultPoller() {
PollerMetadata pollerMetadata = new PollerMetadata();
pollerMetadata.setErrorHandler(t -> log.error("Failed to retrieve data from FTP: {}", t.getMessage(), t));
pollerMetadata.setTrigger(new PeriodicTrigger(60, TimeUnit.SECONDS));
return pollerMetadata;
}
Most likely the thread is still hanging on the read - if you pull the cable from the actual adapter on the computer, the network stack should notify the java process that the socket is gone, but if you pull the cable from a downstream router, there may be no signal. jstack will show what the thread is doing.
You need to set timeouts on the session factory - see the documentation and the FtpClient javadocs - the dataTimeout is used for reads.
I am using spring boot as per examples for TcpInboundGateway,so different devices send data to this Gateways,things works fine but in between in logs it showing following exception:
2015-12-29 18:42:19.455 ERROR 3465 --- [ool-3-thread-47] o.s.i.i.tcp.connection.TcpNetConnection : Read exception 106.221.159.216:38170:8765:934c050d-c4b5-4466-98ab-ee87714c3d00 SocketException:Connection reset
If this exception is resetting connection then how to avoid this reset?What is the cause of this error?
My code as follows
#SpringBootApplication
#IntegrationComponentScan
public class SpringIntegrationApplication extends SpringBootServletInitializer{
public static void main(String[] args) throws IOException {
ConfigurableApplicationContext ctx = SpringApplication.run(SpringIntegrationApplication.class, args);
System.in.read();
ctx.close();
}
#Override
protected SpringApplicationBuilder configure(SpringApplicationBuilder application) {
return application.sources(SpringIntegrationApplication.class);
}
private static Class<SpringIntegrationApplication> applicationClass = SpringIntegrationApplication.class;
#Bean
TcpNetServerConnectionFactory cf(){
TcpNetServerConnectionFactory connectionFactory=new TcpNetServerConnectionFactory(8765);
return connectionFactory;
}
#Bean
TcpInboundGateway tcpGate(){
TcpInboundGateway gateway=new TcpInboundGateway();
gateway.setConnectionFactory(cf());
gateway.setRequestChannel(requestChannel());
return gateway;
}
#Bean
public MessageChannel requestChannel(){
return new DirectChannel();
}
#MessageEndpoint
public class Echo {
#ServiceActivator(inputChannel="requestChannel")
public byte[] echo(byte[] in,#SuppressWarnings("deprecation") #Header("ip_address") String ip){
byte[] rawbytes = gosDataSerivce.byteArrayToHex(in,ip);//Process bytes and returns result
return rawbytes;
}
}
}
After setting singleUse to true now exception message is changed slightly.
2015-12-31 06:09:00.481 ERROR 16450 --- [ool-3-thread-10] o.s.i.i.tcp.connection.TcpNetConnection : Read exception 106.221.146.40:9195:8765:1b4755e8-5b0c-44b9-b4e6-b3aacc25e228 SocketException:Connection reset
Use Case:
I have several clients that established GPRS connection to TcpInboundGateWay and sends login packet,our server will reply to this login packet.If client receives server reply to login packet then it will send data packets at regular interval. Server needs to reply to these packet also if server fails to send reply to those data packets then client GPRS connection is terminated and client will try to establish connections again.Let me know if this use case can be handle with TcpInboundGateWay
Network Trace Analysis
General flow of communication between client and server is as follows:Client sends login packet from ip say 106.221.148.165 so at server connection named 106.221.148.165:63430:8765:cc105da2-dae4-494b-af9c-d1ba268f34f1 is created, that client sends subsequent packets from that ip only.So everything works fine,but after some time same client sends its login packet from another ip say 106.221.142.204.And subsequent packets from new ip.But in logs following error comes that for previous connection exception occurred.
2016-01-05 05:16:14.871 ERROR 6819 --- [pool-3-thread-5] o.s.i.i.tcp.connection.TcpNetConnection : Read exception 106.221.148.165:63430:8765:cc105da2-dae4-494b-af9c-d1ba268f34f1 SocketException:Connection reset
I have set singleUse true and I am using spring integration 4.2.1
This message is emitted when the client closes the socket - if your client only sends one message then closes the socket, you can set singleUse to true and it will suppress this message (as long as the socket is closed normally - between messages).
With Spring Integration version 4.2 and later, the message is not emitted on a normal close, even if singleUse is false.