I need to validate uploaded files against a whitelist of allowed extensions (e.g. png, .pdf etc) and also check whether the extension corresponds to the real file content in order to avoid malicious files.
For now file upload is done via fileUpload() directive that parses file metadata and provides file content as a Source[ByteString, Any] which then gets streamed directly to S3 storage:
def uploadFileRoute: Route =
path("files") {
post {
(withSizeLimit(documentSizeLimit) & fileUpload("file")) {
case (fileInfo: FileInfo, contentSource: Source[ByteString, Any]) =>
onSuccess(service.uploadToS3(fileInfo, contentSource)) {
completeDefault(_)
}
}
}
}
Is there any good solution on how to perform such validation and to keep using the streaming upload (e.g. not to store the whole file locally ) if possible? Any code examples are appreciated!
Related
I would like to download a list of files with name and content in Apache Camel.
Currently I am downloading the file content of all files as byte[] and storing them in a List. I then read the list using a ConsumerTemplate.
This works well. This is my Route:
from(downloadUri)
.aggregate(AggregationStrategies.flexible(byte[].class).accumulateInCollection(
LinkedList.class))
.constant(true)
.completionFromBatchConsumer()
.to("direct:" + this.destinationObjectId);
I get the List of all downloaded file contents as byte[] as desired.
I would like to extend it now so that it downloads the content and the file name of each file. It shall be stored in a pair object:
public class NameContentPair {
private String fileName;
private byte[] fileContent;
public NameContentPair(String fileName, byte[] fileContent) { ... }
}
These pair objects for each downloaded file shall in turn be stored in a List. How can I change or extend my Route to do this?
I tried Camel Converters, but was not able to build them properly into my Route. I always got the Route setup wrong.
I solved this by implementing a custom AggregationStrategy.
It reads the file name and the file content from each Exchange and puts them into a list as a NameContentPair objects. The file content and file name is present in the Exchange's body as a RemoteFile and is read from there.
The general aggregation implementation is based on the example implementation from https://camel.apache.org/components/3.15.x/eips/aggregate-eip.html
The aggregation strategy is then added to the route
from(downloadUri)
.aggregate(new FileContentWithFileNameInListAggregationStrategy())
.constant(true)
.completionFromBatchConsumer()
.to("direct:" + this.destinationObjectId);
I am trying to use Cloudinary's .downloadMulti(String tag, Map options) to generate a URL to download multiple images as a zip with the same tag. I am generating the URL seemingly just fine, but when I go to the URL, I am met with {"error":{"message":"Missing required parameter - timestamp"}}.
I've researched a bit and I saw that I need to sign the request, but it isn't saying that I'm missing that - just the timestamp. I believe the request is already being signed, just need a proper timestamp. I believe it needs to be within the constructor, but when I called Util.timestamp() it isn't recognized as a reference.
My Cloudinary initializer:
private final Cloudinary cloudinary = new Cloudinary(ObjectUtils.asMap(
"cloud_name", "dxoa7bbix",
"api_key", "161649288458746",
"api_secret", "..."));
My upload method:
public Photo uploadOrderImage(String imageURL, String publicId, Order order, String photoType) throws IOException {
Map result = cloudinary.uploader().upload(new File(imageURL), ObjectUtils.asMap(
"public_id", publicId,
"tags", order.getId().toString()));
Photo sellOrderPhoto = new Photo(
result.get("secure_url").toString(),
photoType,
order
);
return photoRepository.save(sellOrderPhoto);
}
This is my download method:
public String downloadPhotos(String tag) throws IOException {
return cloudinary.downloadMulti(tag, ObjectUtils.asMap(
"tags", tag
));
}
An example URL my download method returns: Generated URL: https://api.cloudinary.com/v1_1/dxoa7bbix/image/multi?mode=download&async=false&signature=5f5da549fc78ea3fd50f034cdc76e2cce3089d48&api_key=161649288458746&tag=137×tamp=1638583257.
Overall, I think the problem is the lack of a timestamp. If you have any ideas, that would be great!
The reason for the error is that the parameter is expected to be called timestamp but based on the URL you shared, it is actually ×tamp.
If you want to generate a URL to a ZIP file containing assets that share a particular tag, then you will want to use the generate_archive method and not multi which provides different functionality.
If you replace the following code:
return cloudinary.downloadMulti(tag, ObjectUtils.asMap(
"tags", tag
));
With:
return cloudinary.downloadZip(ObjectUtils.asMap(
"tags", tag,
"resource_type", "image")
);
Then that will generate and return a URL to a ZIP file that will get created when the URL is accessed and contain images from your cloud that contain the tag you specified.
The Cloudinary Java SDK you're using will handle the signature/timestamp generation automatically when using any of the built-in methods, therefore, you don't need to make any changes to the SDK code or calculate the signature yourself if using the built-in methods. Signature generation is only necessary if you're not intending to use any of the SDKs but integrate with the Cloudinary API using your own custom code - such as if you're using a language for which a Cloudinary SDK doesn't yet exist. In such cases, if you want to perform authenticated API calls, you will need to generate authentication signatures yourself.
In my case, I wanted to read all XML form my s3bucket/ parsing then move all parsed files to the same s3Bucker/
for me parsing logic is working fine but I am not able to move all files.
this is the example I am trying to use
**s3 = boto3.resource('s3')
src_bucket = s3.Bucket('bucket1')
dest_bucket = s3.Bucket('bucket2')
for obj in src_bucket.objects.all():
filename= obj.key.split('/')[-1]
dest_bucket.put_object(Key='sample/' + filename, Body=obj.get()["Body"].read())**
above code is not working for me at all (I have to give s3 folder full access and for testing given public full access as well).
Thanks
Check out this answer. You could use python endshwith() function and pass ".xml" to it, get a list of those files and copy them to the destination bucket and then delete them from the source bucket.
I need a 100% fail-safe way to determine if a file name/file URL is available for writing data to disk without accidentally deleting an existing file.
Currently I'm using NSFileManager.fileExists(atPath:), but I'm wondering, what if the parent directory is in iCloud and my file URL points to an existing file in iCloud that hasn't been downloaded to the device yet?
What got me thinking are these two methods of URL:
func checkResourceIsReachable() throws -> Bool
func checkPromisedItemIsReachable() throws -> Bool
There is a distinction between a resource and a promised item. A promised item is defined as:
A promised item is not guaranteed to have its contents in the file
system until you use a file coordinator to perform a coordinated read
on its URL, which causes the contents to be downloaded or otherwise
generated.
Background:
What I'm after is an extension for FileManager that returns the next available, non-existing file name for a given (existing) file URL. For example:
Let's say my existing document is: /documents/Text.txt
The next available file name could be: /documents/Text 2.txt
How can I check that Text 2.txt really doesn't exist locally and in iCloud?
Here's my current approach:
// Safer than: FileManager.default.fileExists(at: url) ?
private func fileExists(at url:URL) throws -> Bool
{
if (try url.resourceValues(forKeys: [.isUbiquitousItemKey])).isUbiquitousItem == true
{
return try url.checkPromisedItemIsReachable()
}
else
{
return try url.checkResourceIsReachable()
}
}
I'm trying to allow users to upload photos to the server and then view them (all users can view all photos) in production (NOT development). While in development mode everything is simple - I can upload the files to the public folder and then read then from there, in production mode I don't have access to the public folder anymore (as this approach is for static accesses and not dynamic).
So, I have 2 issues:
Upload: currently I can't understand how to save the uploaded photos to a specific folder, without using an absolute path to the location where I want the photos to be saved.
Here is the code for upload (similarly to this guide - https://www.playframework.com/documentation/2.4.x/ScalaFileUpload):
def uploadPhoto = Action(parse.multipartFormData) { request =>
import play.api.mvc.MultipartFormData
import play.api.libs.Files.TemporaryFile
import java.io.File
import java.nio.file.Path
import java.nio.file.Paths
try {
val multipartForm: MultipartFormData[TemporaryFile] = request.body
val pathDev: Path = Paths.get("./public/img");
val pathProduction: Path = Paths.get("/...<[full path]>.../public/img");
val imgPath =
if (Files.exists(pathDev)) { pathDev.toString() }
else { pathProduction.toString() }
val newFile = img.get.ref.moveTo(new File(s"$imgPath/$imgName"))
// [HERE I save the path to my DB]
Ok(Json.obj("imgName" -> imgName))
} catch {
case e: Exception =>
BadRequest("unknown error")
}
}
It is unclear to me how to serve the uploaded images back to users that want to see them.
I want to dynamically change the scr in the img html tag to show the relevant image, like so: $("#img").attr("src",'assets/img/1.jpg');
But as the public folder is not available, the images are "not there" (at least until I will "stage" the project and re-run it - https://www.playframework.com/documentation/2.4.x/Assets).
I tried the following approach (https://www.playframework.com/documentation/2.4.x/ScalaStream):
I have added the following line to my conf/routes file:
GET /img/:filename controllers.MyController.getPhoto(filename)
and defined the following function in the controller:
def getPhoto(filename: String) = Action {
Ok.sendFile(new java.io.File("./img/" + filename))
}
But the browser is downloading the file instead of showing it...
These are related:
Handling dynamic created files in play 2
How to serve uploaded files in Play!2 using Scala?
Any assistance will be very appropriated.
Here's how I fix this
ISSUE 1
For upload file path, in play you can define configurations in conf/application.conf file, and you can use different file for production mode, using -Dconfig.file=/path/to/the/file.
So I defined an attribuite called myapp.image.base, in debug mode just set it to "", and in production mode (I created a file called conf/application.prod.conf) , I put an absolute path to it.
So in my code, I always use the following command for file path (it's in Java, but you should find a similar way in Scala for reading configuration)
Play.application().configuration().getString("myapp.image.base")+"img"
ISSUE 2
For serving image
You need to create a router.
First in your routes file, add something like this:
GET /user/images/:name controllers.Application.imageAt(name:String)
And write a simple file reader in action imageAt which return the file in stream. Again my sample is in Java but you should archive the same using Scala
File imageFile = new File(ReportFileHelper.getImagePath());
if (imageFile.exists()) {
//resource type such as image+png, image+jpg
String resourceType = "image+"+imageName.substring(imageName.length()-3);
return ok(new FileInputStream(imageFile)).as(resourceType);
} else {
return notFound(imageFile.getAbsoluteFile());
}
After that, the images is reachable from url /user/images/
Play reads files off the classpath (which includes the assets directory). On the JVM, the classpath is immutable—once started, files added to folders on the classpath will not actually be added to the classpath.
This works in development mode because Play reloads the classpath whenever it detects a change. That same hot-reloading is not enabled in production (for good reason).