Thanks for attention, i using Spring Integration in my project, i want to retrieve files from servers into tmp folder by int-ftp:inbound-channel-adapterand move files to orginal forder by int-file:outbound-gateway for future batch processing, but i feel when file name is duplicate int-file:outbound-gateway not working for me and does not transmit the file and seems ignore them, how to solving this my problem.
<int-file:outbound-gateway id="tmp-mover"
request-channel="ready-to-process-inbound-tmp-mover"
reply-channel="ready-to-process-inbound"
directory="${backupRootPath}/ali/in//"
mode="REPLACE" delete-source-files="true"/>
Set the local-filter in the ftp inbound channel adapter to an AcceptAllFileListFilter. By default, it's an AcceptOnceFileListFilter.
Related
Good day,
I am using spring integration 2.2 to build the listener to a folder. May I know how can I check will my listener pick up file when file transferring half way? Or it will wait until the complete file has been transferred?
Let say there is a txt file with 10MB, but due to internet slowness, the file taking 10 seconds to transfer to the folder. Let say at the 5th second, the file only done transfer 5MB, still 5MB to go, possible the listener go to pick up the file and process?
Here is part of my listener code in xml:
<int-file:inbound-channel-adapter id="hostFilesIn" directory="${hostfile.dir.in}"
prevent-duplicates="true" filename-regex="${hostfile.in.filename-regex}">
<int:poller id="poller" fixed-rate="${poller.fixrate:15000}" max-messages-per-poll="1" />
</int-file:inbound-channel-adapter>
Let me know if need more information.
Here is the problem that happen:
From the log, we can saw a file is being added to queue,
2019-02-01 11:13:33.011 [task-scheduler-9] DEBUG org.springframework.integration.file.FileReadingMessageSource - Added to queue: [/apps/NAS/ftp/in/incompleteL041.TXT]
After that hitting the following error:
2019-02-01 11:13:33.096 [task-scheduler-9] DEBUG c.c.c.c.g.a.auditservice.SimpleErrorIdResolver - ERROR MESSAGE7 :
java.io.FileNotFoundException: /apps/NAS/ftp/in/incompleteL041.TXT (A file or directory in the path name does not exist.)
Its because, when the file finish transfer, it will rename it to L041.TXT.
Kindly advise.
Your understanding is fully correct. That's how file system behaves. If file is already there, even if it is still in writing mode, it is visible for all others including the mentioned file reading message source.
To prevent reading non-complete files you really need to implement a logic when non-finished file has a different name and filters on the file inbound channel adapter should be configured the way that they don't see such a file.
Something like this regexp ^(incomplete).*?\.TXT should work for you current explanation.
I have a spring batch integration where multiple servers are polling a single file directory. This causes a problem where a file can be processed up by more than one. I have attempted to add a nio-lock onto the file once a server has got it but this locks the file for processing so it can't read the contents of the file.
Is there a spring batch/integration solution to this problem or is there a way to rename the file as soon as it is picked up by a node?
Consider to use FileSystemPersistentAcceptOnceFileListFilter with the shared MetadataStore: http://docs.spring.io/spring-integration/reference/html/system-management-chapter.html#metadata-store
So, only one instance of your application will be able to pick up a file.
Even if we find a solution for nio-lock, you should understand that lock means "do not touch until freed". Therefore when one instance has done its work, another one is ready to pick up the file. I guess that isn't your goal.
Is there a preserve-timestamp property in the file:inbound-channel-adapter like in the ftp:inbound-channel-adapter or sftp:inbound-channel-adapter?
I have a requirement to copy some files from one folder to another folder (not FTP) and I need to keep the timestamp.
If there isn't such property, can anyone suggest me how to this with spring integration?
For information, I'm using spring-integration 3.0.8.RELEASE.
Thanks a lot for the help.
The payload of the message from the file adapter is a java.io.File. If you want to move it to some other directory on the same physical disk, you could use a simple <service-activator ... expression="payload.renameTo(...)" />.
If you are using the file outbound channel adapter to copy the file to another disk, there is (currently) no option to preserve the timestamp.
I have opened a JIRA issue to add it as a new feature.
In the meantime; you could save off the lastModified in a header (using a header enricher) make the last channel before the outbound adapter a pub/sub channel, and add a second subscriber to set the lastModified on the new file.
We need to delete the files after reading it from the S3 bucket, I could not find a option. How do we delete the source file after reading the file from S3 bucket and how do I backup files after successfully reading it?
Is there way that S3-Inbound-Adapter outputs the payload to a outputchannel?
Appreciate your time and help!
Regards
Karthik
Karthik,
The S3-Inbound-Adapter works like all other file-based adapters for the remote file system. I mean FTP and SFTP. We synchronize the remote directory with the local one and pick up files from there on each poll. So, the remove operation should not be as a part of send file to output-channel process, just because the remote transfer and process are separated over the time.
I only see the option for you to delete/backup file in the end of your reading process, e.g. one more subscriber for the publish-subscriber-channel, or <transaction-synchronization on the <poller> of <int-aws:s3-inbound-channel-adapter>.
But with that you should supply somehow the AmazonS3Object properties to invoke the AmazonS3.deleteObject(String bucketName, String key) in the end...
I'm sure that we will be able to transfer those options through the MessageHeaders, so feel free to raise a JIRA on the matter!
I am not able to retrieve records from flat file using fileadapter ver 5.6 with JMS. It always show this error at console,
Startup error. SDK Error: Could not open JMS shared library jms, DllError.
The error occurred on starting the adapter after initialization. The Repository URL is D:\bala\input\Work\AT_adfiles_53689.dat and the Configuration URL is Fileadapter/FileAdapterConfiguration..
Its working fine with RV but not with JMS. Kindly help me out..
I found the solution to the problem above. First look into the AT_adfiles_xxxxx.tra under your working adapter directory. Look for the line where it said "tibco.env.PATH=xxxxx"
First of all, look into all those bins directory, you will find some of the bin folder actually contain libeay32.dll" and "ssleay32.dll". The problem is where the sdk\5.5\bin contain different version of libeay32.dll" and "ssleay32.dll" to other folder. In order for you to run this correctly, all of libeay32.dll" and "ssleay32.dll" should be in the same version.
So which ever version you decided to use, make a copy of that to other folders that contain the same file. What i did to preserve the original version of those is by renaming the original with .bak at the end.
This should allow you to test the file adapter!