Download file contents and names into a List with Apache Camel FTP - ftp

I would like to download a list of files with name and content in Apache Camel.
Currently I am downloading the file content of all files as byte[] and storing them in a List. I then read the list using a ConsumerTemplate.
This works well. This is my Route:
from(downloadUri)
.aggregate(AggregationStrategies.flexible(byte[].class).accumulateInCollection(
LinkedList.class))
.constant(true)
.completionFromBatchConsumer()
.to("direct:" + this.destinationObjectId);
I get the List of all downloaded file contents as byte[] as desired.
I would like to extend it now so that it downloads the content and the file name of each file. It shall be stored in a pair object:
public class NameContentPair {
private String fileName;
private byte[] fileContent;
public NameContentPair(String fileName, byte[] fileContent) { ... }
}
These pair objects for each downloaded file shall in turn be stored in a List. How can I change or extend my Route to do this?
I tried Camel Converters, but was not able to build them properly into my Route. I always got the Route setup wrong.

I solved this by implementing a custom AggregationStrategy.
It reads the file name and the file content from each Exchange and puts them into a list as a NameContentPair objects. The file content and file name is present in the Exchange's body as a RemoteFile and is read from there.
The general aggregation implementation is based on the example implementation from https://camel.apache.org/components/3.15.x/eips/aggregate-eip.html
The aggregation strategy is then added to the route
from(downloadUri)
.aggregate(new FileContentWithFileNameInListAggregationStrategy())
.constant(true)
.completionFromBatchConsumer()
.to("direct:" + this.destinationObjectId);

Related

Cloudinary Error: {"error":{"message":"Missing required parameter - timestamp"}}

I am trying to use Cloudinary's .downloadMulti(String tag, Map options) to generate a URL to download multiple images as a zip with the same tag. I am generating the URL seemingly just fine, but when I go to the URL, I am met with {"error":{"message":"Missing required parameter - timestamp"}}.
I've researched a bit and I saw that I need to sign the request, but it isn't saying that I'm missing that - just the timestamp. I believe the request is already being signed, just need a proper timestamp. I believe it needs to be within the constructor, but when I called Util.timestamp() it isn't recognized as a reference.
My Cloudinary initializer:
private final Cloudinary cloudinary = new Cloudinary(ObjectUtils.asMap(
"cloud_name", "dxoa7bbix",
"api_key", "161649288458746",
"api_secret", "..."));
My upload method:
public Photo uploadOrderImage(String imageURL, String publicId, Order order, String photoType) throws IOException {
Map result = cloudinary.uploader().upload(new File(imageURL), ObjectUtils.asMap(
"public_id", publicId,
"tags", order.getId().toString()));
Photo sellOrderPhoto = new Photo(
result.get("secure_url").toString(),
photoType,
order
);
return photoRepository.save(sellOrderPhoto);
}
This is my download method:
public String downloadPhotos(String tag) throws IOException {
return cloudinary.downloadMulti(tag, ObjectUtils.asMap(
"tags", tag
));
}
An example URL my download method returns: Generated URL: https://api.cloudinary.com/v1_1/dxoa7bbix/image/multi?mode=download&async=false&signature=5f5da549fc78ea3fd50f034cdc76e2cce3089d48&api_key=161649288458746&tag=137×tamp=1638583257.
Overall, I think the problem is the lack of a timestamp. If you have any ideas, that would be great!
The reason for the error is that the parameter is expected to be called timestamp but based on the URL you shared, it is actually ×tamp.
If you want to generate a URL to a ZIP file containing assets that share a particular tag, then you will want to use the generate_archive method and not multi which provides different functionality.
If you replace the following code:
return cloudinary.downloadMulti(tag, ObjectUtils.asMap(
"tags", tag
));
With:
return cloudinary.downloadZip(ObjectUtils.asMap(
"tags", tag,
"resource_type", "image")
);
Then that will generate and return a URL to a ZIP file that will get created when the URL is accessed and contain images from your cloud that contain the tag you specified.
The Cloudinary Java SDK you're using will handle the signature/timestamp generation automatically when using any of the built-in methods, therefore, you don't need to make any changes to the SDK code or calculate the signature yourself if using the built-in methods. Signature generation is only necessary if you're not intending to use any of the SDKs but integrate with the Cloudinary API using your own custom code - such as if you're using a language for which a Cloudinary SDK doesn't yet exist. In such cases, if you want to perform authenticated API calls, you will need to generate authentication signatures yourself.

Placing file inside folder of S3 bucket

have a spring boot application, where I am tring to place a file inside folder of S3 target bucket. target-bucket/targetsystem-folder/file.csv
The targetsystem-folder name will differ for each file which will be retrived from yml configuration file.
The targetsystem-folder have to created via code if the folder doesnot exit and file should be placed under the folder
As I know, there is no folder concept in S3 bucket and all are stored as objects.
Have read in some documents like to place the file under folder, have to give the key-expression like targetsystem-folder/file.csv and bucket = target-bucket.
But it doesnot work out.Would like to achieve this using spring-integration-aws without using aws-sdk directly
<int-aws:s3-outbound-channel-adapter id="filesS3Mover"
channel="filesS3MoverChannel"
transfer-manager="transferManager"
bucket="${aws.s3.target.bucket}"
key-expression="headers.targetsystem-folder/headers.file_name"
command="UPLOAD">
</int-aws:s3-outbound-channel-adapter>
Can anyone guide on this issue
Your problem that the SpEL in the key-expression is wrong. Just try to start from the regular Java code and imagine how you would like to build such a value. Then you'll figure out that you are missing concatenation operation in your expression:
key-expression="headers.targetsystem-folder + '/' + headers.file_name"
Also, please, in the future provide more info about error. In most cases the stack trace is fully helpful.
In the project that I was working before, I just used the java aws sdk provided. Then in my implementation, I did something like this
private void uploadFileTos3bucket(String fileName, File file) {
s3client.putObject(new PutObjectRequest("target-bucket", "/targetsystem-folder/"+fileName, file)
.withCannedAcl(CannedAccessControlList.PublicRead));
}
I didn't create anymore configuration. It automatically creates /targetsystem-folder inside the bucket(then put the file inside of it), if it's not existing, else, put the file inside.
You can take this answer as reference, for further explanation of the subject.
There are no "sub-directories" in S3. There are buckets and there are
keys within buckets.
You can emulate traditional directories by using prefix searches. For
example, you can store the following keys in a bucket:
foo/bar1
foo/bar2
foo/bar3
blah/baz1
blah/baz2

Spring batch to upload a CSV file and insert into database

My project has this requirement where user uploads a CSV file which has to be pushed to sql server database.
I know we can use Spring batch to process large number of records. But I'm not able to find any tutorial/sample code for this requirement of mine.
All the tutorials which I came across just hardcoded the CSV file name and in-memory databases in it like below:
https://spring.io/guides/gs/batch-processing/
User Input file is available in shared drive location on schduled time with file name prefix as eg: stack_overlfow_dd-MM-yyyy HH:mm, on daily basis how can I poll the Network shared drive for every 5-10 minutes atleast for one hour daily if its matches with regex then upload to database.
How can I take the csv file first from shared location and store it in memory or somewhere and then configure spring batch to read that as input.
any help here would be appreciated. Thanks In advance
All the tutorials which I came across just hardcoded the CSV file name and in-memory databases
You can find samples in the official repo here. Here is an example where the input file name is not hardcoded but passed as a job parameter.
How can I take the csv file first from shared location and store it in memory or somewhere and then configure spring batch to read that as input.
You can proceed in two steps: download the file locally then read/process/write it to the database (See https://stackoverflow.com/a/52110781/5019386).
how can I poll the Network shared drive for every 5-10 minutes atleast for one hour daily if its matches with regex then upload to database.
Once you have defined your job, you can schedule it to run when you want using:
a scheduler like Quartz
or using Spring's task scheduling features.
or using a combination of Spring Integration and Spring Batch. Spring integration would poll the directory and then launches a Spring Batch job when appropriate. This approach is described here.
More details on job scheduling here.
You can make a service layer that can process excel file and read data from file and construct java object to save into DB. Here I have used apache POI to parse Excel data and read from excel sheet.
public class FileUploadService {
#Autowired
FileUploadDao fileUploadDao;
public String uploadFileData(String inputFilePath) {
Workbook workbook = null;
Sheet sheet = null;
try {
workbook = getWorkBook(new File(inputFilePath));
sheet = workbook.getSheetAt(0);
/*Build the header portion of the Output File*/
String headerDetails = "EmployeeId,EmployeeName,Address,Country";
String headerNames[] = headerDetails.split(",");
/*Read and process each Row*/
ArrayList < ExcelTemplateVO > employeeList = new ArrayList < > ();
Iterator < Row > rowIterator = sheet.iterator();
while (rowIterator.hasNext()) {
Row row = rowIterator.next();
//Read and process each column in row
ExcelTemplateVO excelTemplateVO = new ExcelTemplateVO();
int count = 0;
while (count < headerNames.length) {
String methodName = "set" + headerNames[count];
String inputCellValue = getCellValueBasedOnCellType(row, count++);
setValueIntoObject(excelTemplateVO, ExcelTemplateVO.class, methodName, "java.lang.String", inputCellValue);
}
employeeList.add(excelTemplateVO);
}
fileUploadDao.saveFileDataInDB(employeeList);
} catch (Exception ex) {
ex.printStackTrace();
}
return "Success";
}
I believe your question have already been answered here.
The author of the question has even uploaded a repository of his working result :
https://github.com/PriyankaBolisetty/SpringBatchUploadCSVFileToDatabase/tree/master/src/main/java/springbatch_example
You can retrieve and filter files' lists in a shared drive using JCIFS API method SmbFile.listFiles(String wildcard).

Custom file name while keeping extension

Following this part of the docs:
https://laravel.com/docs/5.3/filesystem#file-uploads
I'm trying to store a file (an image) and specify a custom file name, but I don't want to change the extension.
If I do:
$request->file('avatar')->storeAs('avatars', 'custom_file_name');
The file will save as custom_file_name (with no extension), rather than custom_file_name.png).
How can I specify a custom file name while keeping the original file extension?
You can get extension of uploaded file with:
$extension = $request->photo->extension();
And apply it manually to the stored file name.
The UploadedFile class also contains methods for accessing the file's fully-qualified path and its extension. The extension method will attempt to guess the file's extension based on its contents.
https://laravel.com/docs/5.4/requests#files
The code below checks if request has a file by id/name avatar and then stores it with custom_name.{uploaded_files_extension}
if($request->hasFile('avatar')) {
$imgfile = Input::file('avatar');
$custom_name = 'custom_name'.$imgfile->getClientOriginalExtension();
$request->file('avatar')->storeAs('avatars', $custom_name);
}
Just to buttress #alexey-mezenin answer,
you might have issue adding the extension
//get extension
$extension = $request->file('featured_img')->extension();
$request->file('featured_img')->storeAs('/public/album',$post->title.'.'.$extension);
With that you can have a complete code stored in your /storage/public/album

Play Framework: Handling dynamic created files (images) in PRODUCTION mode

I'm trying to allow users to upload photos to the server and then view them (all users can view all photos) in production (NOT development). While in development mode everything is simple - I can upload the files to the public folder and then read then from there, in production mode I don't have access to the public folder anymore (as this approach is for static accesses and not dynamic).
So, I have 2 issues:
Upload: currently I can't understand how to save the uploaded photos to a specific folder, without using an absolute path to the location where I want the photos to be saved.
Here is the code for upload (similarly to this guide - https://www.playframework.com/documentation/2.4.x/ScalaFileUpload):
def uploadPhoto = Action(parse.multipartFormData) { request =>
import play.api.mvc.MultipartFormData
import play.api.libs.Files.TemporaryFile
import java.io.File
import java.nio.file.Path
import java.nio.file.Paths
try {
val multipartForm: MultipartFormData[TemporaryFile] = request.body
val pathDev: Path = Paths.get("./public/img");
val pathProduction: Path = Paths.get("/...<[full path]>.../public/img");
val imgPath =
if (Files.exists(pathDev)) { pathDev.toString() }
else { pathProduction.toString() }
val newFile = img.get.ref.moveTo(new File(s"$imgPath/$imgName"))
// [HERE I save the path to my DB]
Ok(Json.obj("imgName" -> imgName))
} catch {
case e: Exception =>
BadRequest("unknown error")
}
}
It is unclear to me how to serve the uploaded images back to users that want to see them.
I want to dynamically change the scr in the img html tag to show the relevant image, like so: $("#img").attr("src",'assets/img/1.jpg');
But as the public folder is not available, the images are "not there" (at least until I will "stage" the project and re-run it - https://www.playframework.com/documentation/2.4.x/Assets).
I tried the following approach (https://www.playframework.com/documentation/2.4.x/ScalaStream):
I have added the following line to my conf/routes file:
GET /img/:filename controllers.MyController.getPhoto(filename)
and defined the following function in the controller:
def getPhoto(filename: String) = Action {
Ok.sendFile(new java.io.File("./img/" + filename))
}
But the browser is downloading the file instead of showing it...
These are related:
Handling dynamic created files in play 2
How to serve uploaded files in Play!2 using Scala?
Any assistance will be very appropriated.
Here's how I fix this
ISSUE 1
For upload file path, in play you can define configurations in conf/application.conf file, and you can use different file for production mode, using -Dconfig.file=/path/to/the/file.
So I defined an attribuite called myapp.image.base, in debug mode just set it to "", and in production mode (I created a file called conf/application.prod.conf) , I put an absolute path to it.
So in my code, I always use the following command for file path (it's in Java, but you should find a similar way in Scala for reading configuration)
Play.application().configuration().getString("myapp.image.base")+"img"
ISSUE 2
For serving image
You need to create a router.
First in your routes file, add something like this:
GET /user/images/:name controllers.Application.imageAt(name:String)
And write a simple file reader in action imageAt which return the file in stream. Again my sample is in Java but you should archive the same using Scala
File imageFile = new File(ReportFileHelper.getImagePath());
if (imageFile.exists()) {
//resource type such as image+png, image+jpg
String resourceType = "image+"+imageName.substring(imageName.length()-3);
return ok(new FileInputStream(imageFile)).as(resourceType);
} else {
return notFound(imageFile.getAbsoluteFile());
}
After that, the images is reachable from url /user/images/
Play reads files off the classpath (which includes the assets directory). On the JVM, the classpath is immutable—once started, files added to folders on the classpath will not actually be added to the classpath.
This works in development mode because Play reloads the classpath whenever it detects a change. That same hot-reloading is not enabled in production (for good reason).

Resources