How to configure the dynamic file name in Spring Integration flow?
Scenario: On FTP, every day file get generated for current date. Through the SFTP, need to read current date file.
Example: *_20220525_DATA.TXT, *_20220526_DATA.TXT
Why just patternFilter("*_DATA.TXT") doesn't work for you?
The SftpSimplePatternFileListFilter cannot be modified for its pattern.
Although you can implement a custom FileListFilter to have its patterns to be changed every day.
Related
How to configure the dynamic file name in Spring Integration flow?
Scenario: On FTP, every day file get generated for current date. Through the SFTP, need to read current date file.
Example: *_20220525_DATA.TXT, *_20220526_DATA.TXT
Why just patternFilter("*_DATA.TXT") doesn't work for you?
The SftpSimplePatternFileListFilter cannot be modified for its pattern.
Although you can implement a custom FileListFilter to have its patterns to be changed every day.
I have a pre existing springboot application like this one which uses spring batch to read a huge CSV file(size=1GB) from a local location and inserts the content after some changes to a database. Now, the file will be coming from a bucket in Google Cloud Platform(GCP). I am planning to use something like Inbound streaming channel adapter mentioned here. Is this feasible?If yes, what changes will be needed in the ItemReader of the batch? Or is there any other option I need to consider? Any suggestion is welcome.
The FlatFileItemReader works with any Resource implementation. So you could use an URLResource and point it to your file on GCP without having to download it.
Here’s a specific example for a CSV file using the URLResource:
URLResource ur = new URLResource("http://www.dukelearntoprogram.com/course2/java/food.csv");
for (CSVRecord record : ur.getCSVParser()) {
// print or process fields in record
String name = record.get("Name");
// other processing
}
This was extracted from non-official documentation.
by adding relevant dependencies and changing the local file path
->"flatFileItemReader.setResource(new FileSystemResource("src/main/resources/message.csv"));"
to this
->"flatFileItemReader.setResource(new UrlResource("storage.googleapis.com/test-mybucket/mycsv.csv"));"
will it work ??
I have a scenario where one of the test REST API returns a transaction_Id, this transaction Id I need to pass in my next request(API Request), also upload the file as well. This upload file has name format like ${transaction_Id}_1_1_bkg.raw,without correct format backend server may not be able to analyze the data.
Since I have to run a load test, these transaction Ids will generate at run time and the same I have to change the file name and upload it as well.
Did any one had face this challenge before in Jmeter to change the file name at run time and upload it, if yes, what was the solution.
Thanks,
Akshat
You could copy the file to a new one with the desired name using JSR223 PreProcessor like
org.apache.commons.io.FileUtils.copyFile(new File('template_bkg.raw'), new File(vars.get('transaction_Id') + '_1_1_bkg.raw'))
where vars stands for JMeterVariables class instance, see the JavaDoc for more information on all existing functions and Top 8 JMeter Java Classes You Should Be Using with Groovy article to learn more about JMeter API shorthands available for the JSR223 Test Elements.
Once upload is finished you can delete the file using JSR223 PostProcessor
I have two api for file download and upload. (/static/{filename} api for downloading, /upload api for uploading) For consistency, it need a test file to ensure the functionality of them two. I check the official fs_test.go, but it is too massive to use, is there a trick way to do it?
Do you need a unit test? If you are supposed to test saving the file you can use a brute force.
Create file
Check if it exists (optional: check the file data)
Delete file
We are using the push notifications provided by google Drive API. The change id is pushed to the service by google, and we use the GET Api to retrive the change object. Our requirement is to find from the change object whether it is a file upload/update or trash. We tried using created date, modified date and modification date to find the difference. But the modification date is not providing the time the change happened originally and there is a difference of 2 seconds sometimes.. Also, When a folder is shared to multiple users, for the same change, the modification date which each user receive is different from that of the other and the difference is in seconds.
Please suggest if there is a way to distinguish between file upload/update/trash from change object.
Also, provide inputs on why the modification date is different for each user, and different from file modified date.
NOTE: We cannot use watch for files as per our requirement as there are lot of files in the drive and we can watch only for changes in the entire drive.