Download one file at a time through the same session in Apache Camel FTP - ftp

I want to implement following use case with Apache Camel FTP:
On a remote location I have 0 to n amount of files stored.
When I receive a command, using FTP, I want to download one file as a byte array (which one does not matter), if any files are available.
When the file is downloaded, I want to save it in a database as a blob.
Then I want to delete the stored/processed file on the remote location
Wait for the next download command and once received go back to step 1.
The files have to be downloaded through the same FTP session.
My problem is that if I use a normal FTP route, it downloads all available files.
When I tell the route to only download one, I have to create a new route for the other files and I cannot reuse the FTP session.
Is there a way to implement this use case with Apache Camel FTP?

Camel-ftp doesn't consume all available files at once it consumes them individually one after another meaning that each file gets processed separately. If you need to process them in some specific order you can try using file-name or modified date with sortBy option.
If you want to control when file gets downloaded i.e when command gets called you can call FTP Consumer endpoint using pollEnrich
Example:
// 1. Loads one file from ftp-server with timeout of 3 seconds.
// 2. logs the body and headers
from("direct:example")
.pollEnrich("ftp:host:port/directoryName", 3000)
.to("log:loggerName?showBody=true&showHeaders=true");
You can call the direct consumer endpoint with ProducerTemplate you can obtain from CamelContext or change it to whatever consumer endpoint fits your use case.
If you need to use dynamic URI you can use simple to provide the URI for poll-enrich and also also provide timeout afterwards.
from("direct:example")
.pollEnrich()
.simple("ftp:host:port/directoryName?fileName=${headers.targetFile}")
.timeout(3000)
.to("log:loggerName?showBody=true&showHeaders=true");

Related

NiFi putFTP not efficient

I have a nifi flow that sends more than 50 files per minute using the putFTP processor. The server has limited resources, but I need to send in a faster pace. I looked at the ftp server logs (not nifi), and my conclusion:
A new ftp connection (session) is created for every file. Is there an option to configure many files on one session? (connect to port 21, authenticate once, and then send many files on different ports)
When sending one file, many CWD (Change Working Directory) commands are sent. For example, sending file to /myfiles/test/dest/file.txt:
CWD /
CWD /myfiles
CWD /
CWD /myfiles/test
CWD /
CWD /myfiles/test/dest
This is not efficient. Is there any way to improve the putFTP? Is this a bug?
First question: use run duration
A new ftp connection (session) is created for every file. Is there an
option to configure many files on one session? (connect to port 21,
authenticate once, and then send many files on different ports)
First, (if it fits your use case) you can use the MergeContent processor to merge multiple (smaller) flow files into one (bigger) flow file and feed it to PutFTP.
Second, the PutFTP processor has the SupportsBatching annotation:
Marker annotation a Processor implementation can use to indicate that
users should be able to supply a Batch Duration for the Processor. If
a Processor uses this annotation, it is allowing the Framework to
batch ProcessSessions' commits, as well as allowing the Framework to
return the same ProcessSession multiple times...
Source: https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/annotation/behavior/SupportsBatching.java
Increase the run duration of your PutFTP processor towards more throughput to use the same task for many flow files. You might want to adjust the Maximum Batch Size in the properties tab to accommodate to that change.
Read more about it here:
Dcoumentation: Run duration
Understanding NiFi processor's "Run Duration" functionality.
What should be Ideal Run-duration and Run schedule configuration in nifi processors
Second question: inspect source code
When sending one file, many CWD (Change Working Directory) commands
are sent. For example, sending file to /myfiles/test/dest/file.txt
By inspecting FTPTransfer.java you can see, that the put method does the following:
put -> get client
put -> get client -> resetWorkingDirectory -> changeWorkingDirectory(homeDirectory)
put -> setAndGetWorkingDirectory
This might be the behavior you discovered.

How to check whether a file downloaded from a ftp location is completed or not using apache camel

I need to check whether the file downloaded from a ftp location using camel is completed, being downloaded or the connection broke. If completed I need to do a certain function else I need to make another call
The FTP component also has the capabilities of File2. If you check the docs you'll see options for preMove, move, and moveFailed. You can change the names of files and/or change the location of files before processing, after successful processing, and after failure.

Using C# Windows service

I would like to create a utility to automatically copy files from one folder to another folder.
• The target and destination folder will be specified as a parameter
• The file type in (i.e. “.csv” or “.txt”) will also be specified as a parameter
How to pass parameters to windows service.. Please reply ..
Set your service to be a web service on some obscure port. Use a REST api (or whatever you like) to put a json/xml (or use get query strings if you like to) post to the service with directory, extension, and whatever else you might need.
The service has the data and can do its thing. You can also define other apis the web server can respond to which can stop the copy process or report status on ongoing copies (10 of 15 files copied, etc). You can enhance and augment it however you like.

how to delete file from the server after downloading it ?or how can i store the file to client machine directly from output stream?

am using liferay custom portlet and in that am using jasper report now my problem is that how can i download the pdf report directly on the client machine
right now am storing the file at server first.then provide url for downloading the pdf to user.but how can i directly store the file to client machine if i have pdf file's outputstream .
ot if i can know some how when user click on the download link and after downloading the file if i want to delete the donlowded file from the server then how can i do it.?if any one can guide me...
I'm not sure what you're asking for is possible, but I would be interested in seeing someone correct that statement though.
Servers really shouldn't be directly storing files on a client machine as that violates the intent of the client server relationship. A client has to make a request for the file and then the client can save that file (eg like a ftp download). Servers just don't manipulate client machines as they see fit.
As far as knowing when a file is downloaded, there isn't anything in a portlet you can do to detect that. You can use ResourceRequest and serveResource method to serve a file, but nothing in the portlet API will inform your portlet that the download is complete or that it wasn't interrupted by something.
As an alternative you might try simply having a cron job that will clean out old files. In this case, make sure to inform users how long they have to successfully download the file.

How to receive a specific file thru BizTalk FTP receive port

My orchestration receives a message that contains a file name and I want to pick that file from my FTP. I can configure FTP receive port to receive all files from some folder in the FTP, but how do I receive a file with specific name?
The process looks like
I would rather recommend your solution of writing a custom .NET component which will fetch the file from FTP location (you can call that component from your expression shape).
Dynamically creating Receive Ports/Receive locations and later removing them won't scale and possibly will get you into serious trouble.
I'm not sure if this link helps - specifically the CreateFtpReceiveLocation method - i.e. programatically adding a receive location (pseudo dynamic receive location).
You would also need to remove the location afterwards. I'm guessing that you can also set the FileMask on the Transport Properties of the 'dynamic' Location to the filename in the Custom Props of the TransportTypeData, and would need to remove the Location once you are done with the file.

Resources