I have a Multipartfile containing a zip thats getting uploaded to spring, after upload the archive should be decompressed and all files of the archive uploaded to a bucket, after validation of the items in Spring.
What is the best practice part to do this? I try to avoid to temporary store those items in the filesystem at the spring service.
What I do at the moment (partly showing):
ZipInputStream zis = new ZipInputStream(multipartfile.stream());
for (ZipEntry entry; (entry = zis.getNextEntry()) != null; ) {
ByteArrayOutputStream contentByteArray = new ByteArrayOutputStream();
while (-1 != (read = zis.read())) {
contentByteArray.write(read);
}
byte[] file = contentByteArray.toByteArray();
validate(file);
uploadToBucket(file);
}
However, I know that is not the solution how to do this smartly. Any suggestions how to do that?
Related
I have a Spring boot API and one of the endpoints allows users to upload video's. Now My controller basically takes the file as a MultiPart file and then I store it in a temp folder accessible to tomcat. Once I have it stored on Disk, I then push the video to an S3 bucket.
Now to me anyway, this seems to be less than optimal, Like if I wanted to have a 100 or a 1000 users upload at once it seems really non performant to write the files to disk first.
As a little background I'm storing it on disk with the intention that if there is a issue pushing to S3 I can retry
The below code might show what I'm doing better than the above:
public Video addVideo(#RequestParam("title") String title,
#RequestParam("Description") String Description,
#RequestParam(value = "file", required = true) MultipartFile file) {
this.amazonS3ClientService.uploadFileToS3Bucket(file, title, description));
}
Method for storing Video file:
String fileNameWithExtenstion = awsS3FileName + "." + FilenameUtils.getExtension(multipartFile.getOriginalFilename());
//creating the file in the server (temporarily)
File file = new File(tomcatTempDir + fileNameWithExtenstion);FileOutputStream fos = new FileOutputStream(file);
fos.write(multipartFile.getBytes());
fos.close();PutObjectRequest putObjectRequest = new PutObjectRequest(this.awsS3Bucket, awsS3BucketFolder + UnigueId + "/" + fileNameWithExtenstion, file);
if (enablePublicReadAccess) {
putObjectRequest.withCannedAcl(CannedAccessControlList.PublicRead);
}
// Upload a file as a new object with ContentType and title
specified.amazonS3.putObject(putObjectRequest);
//removing the file created in the server
file.delete();
So my question is....is there a better way in Tomcat to:
A) Take in a file via a controllerB) Push to S3
There is no other way to do it with multipart. The problem with multipart that to properly segement parts from the requst they need sometimes skipped or be repeatable. That is impossible within memory w/o having memory to explode. Therefore, Commons FileUpload caches them on disk after a certain threshold is reached.
Multipart requests are the worst way for that. I highly recommend to use either PUT or POST with content type application/octet-stream. You can take the bare request input stream and pass to HttpClient to stream to your backend server. I did this already 5 years ago and it works for gigabytes. I have posted the solution in the Apache HttpClient mailing list.
There is one possibility how this could work under specific conditions:
All parts are in the correct physical order you want to read
Your write to a backend is fast enough to sustain the read from the front
Consume the root part and then go over to the next physical one, process the request body lazily. JAX-WS RI (Metro) has a very nice handling of multipart requests for XOP/MTOM. Learn from that because you won't be able to make it any better.
Perhaps you can try to direct stream the input stream from your MultipartFile to S3.
Consider the following uploadFileToS3Bucket method:
public PutObjectResult uploadFileToS3Bucket(InputStream input, long size, String title, String description) {
// Indicate the length of the information to avoid the need to compute it by the AWS SDK
// See: https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/PutObjectRequest.html#PutObjectRequest-java.lang.String-java.lang.String-java.io.InputStream-com.amazonaws.services.s3.model.ObjectMetadata-
ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.setContentLength(size); // rely on Spring implementation. Maybe you probably also can use input.available()
// compute the object name as appropriate
String key = "...";
PutObjectRequest putObjectRequest = new PutObjectRequest(
this.awsS3Bucket, key, input, objectMetadata
);
// The rest of your code
if (enablePublicReadAccess) {
putObjectRequest.withCannedAcl(CannedAccessControlList.PublicRead);
}
// Upload a file as a new object with ContentType and title
return specified.amazonS3.putObject(putObjectRequest);
}
Of course, you need to provide the service the input stream obtained from the client request associated with the MutipartFile object:
public Video addVideo(
#RequestParam("title") String title,
#RequestParam("Description") String Description,
#RequestParam(value = "file", required = true) MultipartFile file) {
try (InputStream input = file.getInputStream()) {
this.amazonS3ClientService.uploadFileToS3Bucket(input, file.getSize(), title, description));
}
}
Probably you can also play with the getBytes method of MultipartFile and create a ByteArrayInputStream to perform the operation.
In addVideo:
byte[] bytes = file.getBytes();
In uploadFileToS3Bucket:
ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.setContentLength(bytes.length);
PutObjectRequest putObjectRequest = new PutObjectRequest(
this.awsS3Bucket, key, new ByteArrayInputStream(bytes), objectMetadata
);
I would prefer the first solution, but try to determine which option offers you the best performance.
I am trying to copy image into assets folder inside WEB-INF folder. Following code successfully copy images outside the project but can't copy inside WEB-INF folder.
public static void copyFile(String source, String destination) throws IOException {
try {
File sourceFile = new File(source);
File destinationFile = new File(destination);
FileInputStream fileInputStream = new FileInputStream(sourceFile);
FileOutputStream fileOutputStream = new FileOutputStream(destinationFile);
int bufferSize;
byte[] bufffer = new byte[512];
while ((bufferSize = fileInputStream.read(bufffer)) > 0) {
fileOutputStream.write(bufffer, 0, bufferSize);
}
fileInputStream.close();
fileOutputStream.close();
} catch (IOException e) {
throw new IOException(e.getMessage());
}
}
I get a image path from Http request.
CopyFile.copyFile(imageUrl, "http://localhost:8080/M.S.-Handloom-Fabrics/static/"+imageName+".png");
I have mapped the resources in dispatcher-servlet.xml
<mvc:resources mapping="/static/**" location="/WEB-INF/assets/"/>
Here is the error
Info: http:\localhost:8080\M.S.-Handloom-Fabrics\static\TueJun1216_27_54NPT20180.png (The filename, directory name, or volume label syntax is incorrect)
http://localhost:8080/M.S.-Handloom-Fabrics/static/"+imageName+".png"
is a URL, not a file-path, and is therefore meaningless as a parameter to the File constructor.
IF you configured your app server to explode your webapp on deploy then you could use ServletContext.getRealPath but as this SO post details nicely you most likely do not want to do this as your saved files will be lost upon re-deploy.
Saving them outside of the web app is the way to go.
I'm working on an image manager system for my Spring MVC project, with the basic functions of displaying the gallery of all images stored in the local image folder, deleting images and uploading new images.
I would like that, once a new image is uploaded, the page reloads the images gallery, including the image just added. What it happens in fact is that the new image is correctly saved on the HD, but it doesn't automatically show up in the resources/img folder in the Java project; therefore, once the page is reloaded, the new image is not there. Only when I manually refresh the project, the new image appears in the resources/img folder.
The odd thing is, I don't have the same problem with the delete method: once an image is deleted, it disappears from the HD AND the resources/img folder, and the page reloads the gallery without showing the image just deleted.
Any idea where the problem could be?
Here is my controller
#Controller
public class imagesManagerController {
// READ ALL FILES FROM IMG FOLDER
#RequestMapping(value = "/imagesManager", method = RequestMethod.GET)
public ModelAndView readImages
(#RequestParam(value = "error", required = false) String error) {
// create model and link it to jsp imagesManager
ModelAndView model = new ModelAndView("imagesManager");
// return content from images folder and add it to model
File imgsPath = new File("C:/Users/Alessandro/workspace/SpringMVCBlog/WebContent/resources/img");
String[] imgsNames = imgsPath.list();
model.addObject("imgsNames", imgsNames);
//if upload fails, display error message
if (error != null) {
model.addObject("error",
"Please select a file to upload");
}
return model;
}
//UPLOAD FILE TO HD
#RequestMapping(value = "/imagesManager/upload", method = RequestMethod.POST)
public String handleFileUpload (#RequestParam("file") MultipartFile file) {
//get img name
String imgName = file.getOriginalFilename();
System.out.println(imgName);
//create file path
String folder = "C:/Users/Alessandro/workspace/SpringMVCBlog/WebContent/resources/img/";
File path = new File (folder+imgName);
System.out.println(path);
if (!file.isEmpty()) {
try {
//get bytes array from file
byte[] bytes = file.getBytes();
//create output stream
BufferedOutputStream stream = new BufferedOutputStream(
new FileOutputStream(path));
//write img content on path
stream.write(bytes);
//close stream
stream.close();
//if upload is successful, reload page
return "redirect:/imagesManager";
} catch (Exception e) {
return "You failed to upload " + imgName + " => " + e.getMessage();
}
} else {
return "redirect:/imagesManager?error";
}
}
// DELETE FILE FROM HD
#RequestMapping(value = "/imagesManager/delete", method = RequestMethod.POST)
public String deleteFile(#RequestParam (value="imgName") String imgName) {
//create file path to be deleted
String folder = "C:/Users/Alessandro/workspace/SpringMVCBlog/WebContent/resources/img/";
File path = new File (folder+imgName);
// delete file
if (path.delete()) {
//if delete is successful, reload page
return "redirect:/imagesManager";
} else {
return "Delete operation failed";
}
}
}
The problem is in the path:
WebContent/resources/img
It is refreshing probably due to IDE server auto-deployment. Test with %TEMP% path and check.
1) You should not save uploaded files to the application server file system.
2) You should not save uploaded files to the application folder as it is part of the deployment. It will only be deployed once and that folder is only for the application files.
Instead, use the cloud or a dedicated file system.
My requirement is to use Web API to send across the network, a zip file (consisting a bunch of files in turn) which should not be written anywhere locally (not written anywhere on the server/client disk). For zipping, I am using DotNetZip - Ionic.Zip.dll
Code at Server:
public async Task<IHttpActionResult> GenerateZip(Dictionary<string, StringBuilder> fileList)
{
// fileList is actually a dictionary of “FileName”,”FileContent”
byte[] data;
using (ZipFile zip = new ZipFile())
{
foreach (var item in filelist.ToArray())
{
zip.AddEntry(item.Key, item.Value.ToString());
}
using (MemoryStream ms = new MemoryStream())
{
zip.Save(ms);
data = ms.ToArray();
}
}
var result = new HttpResponseMessage(HttpStatusCode.OK);
MemoryStream streams = new MemoryStream(data);
//, 0, data.Length-1, true, false);
streams.Position = 0;
//Encoding UTFEncode = new UTF8Encoding();
//string res = UTFEncode.GetString(data);
//result.Content = new StringContent(res, Encoding.UTF8, "application/zip");
<result.Content = new StreamContent(streams);
result.Content.Headers.ContentType = new MediaTypeHeaderValue("application/zip");
//result.Content.Headers.ContentLength = data.Length;
result.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment");
result.Content.Headers.ContentDisposition.FileName = "test.zip";
return this.Ok(result);
}
The issue I am facing is that after the zip file downloaded at client end when modified as a test.bin has its stream contents (byte[] data in this example’s contents) missing. (I am getting back a test.zip file. When I change the file locally from test.zip to test.bin, I am seeing that the File’s contents as shown below. It does not contain the Response.Content values. P.S. I have also tried the MIME type “application/octet-stream” as well. No luck!)
Test.zip aka test.bin’s contents:
{"version":{"major":1,"minor":1,"build":-1,"revision":-1,"majorRevision":-1,"minorRevision":-1},
"content":{"headers":[{"key":"Content-Type","value":["application/zip"]},
{"key":"Content-Disposition","value":["attachment; filename=test.zip"]}]},
"statusCode":200,"reasonPhrase":"OK","headers":[],"isSuccessStatusCode":true}
Can someone please help me on how we can set result.Content with a MemoryStream object (I have seen example of “FileStream” at other places on google to set “result.Content” but I want to use MemoryStream object only!). I am highlighting this because I think the problem lies with setting the MemoryStream object to the result.Content (in order to properly save the streams content into the result.Content object)
P.S. I have also gone thru Uploading/Downloading Byte Arrays with AngularJS and ASP.NET Web API (and a bunch of other links) but it did not help me much… :(
Any help is greatly appreciated. Thanks a lot in advance :)
I got my issue solved!!
All I did was to change the Response Type to HttpResponseMessage and use "return result" in the last line rather than Ok(result) { i.e. HttpResponseMessage Type rather than OKNegiotatedContentResult Type)
How can I get a pdf located in a file in a server's directory structure to load in a browser for users of a Spring MVC application?
I have googled this and found postings about how to generate PDFs, but their answers do not work in this situation. For example, this other posting is not relevant because res.setContentType("application/pdf"); in my code below does not solve the problem. Also, this other posting describes how to do it from a database but does not show full working controller code. Other postings had similar problems that caused them not to work in this case.
I need to simply serve up a file (not from a database) and have it been viewable by a user in their browser. The best I have come up with is the code below, which asks the user to download the PDF or to view it in a separate application outside the browser. What specific changes can I make to the specific code below so that the user automatically sees the PDF content inside their browser when they click on the link instead of being prompted to download it?
#RequestMapping(value = "/test-pdf")
public void generatePdf(HttpServletRequest req,HttpServletResponse res){
res.setContentType("application/pdf");
res.setHeader("Content-Disposition", "attachment;filename=report.pdf");
ServletOutputStream outStream=null;
try {
BufferedInputStream bis = new BufferedInputStream(
new FileInputStream(new File("/path/to", "nameOfThe.pdf")));
/*ServletOutputStream*/ outStream = res.getOutputStream();
//to make it easier to change to 8 or 16 KBs
int FILE_CHUNK_SIZE = 1024 * 4;
byte[] chunk = new byte[FILE_CHUNK_SIZE];
int bytesRead = 0;
while ((bytesRead = bis.read(chunk)) != -1) {outStream.write(chunk, 0, bytesRead);}
bis.close();
outStream.flush();
outStream.close();
}
catch (Exception e) {e.printStackTrace();}
}
Change
res.setHeader("Content-Disposition", "attachment;filename=report.pdf");
To
res.setHeader("Content-Disposition", "inline;filename=report.pdf");
You should also set the Content Length
FileCopyUtils is handy:
#Controller
public class FileController {
#RequestMapping("/report")
void getFile(HttpServletResponse response) throws IOException {
String fileName = "report.pdf";
String path = "/path/to/" + fileName;
File file = new File(path);
FileInputStream inputStream = new FileInputStream(file);
response.setContentType("application/pdf");
response.setContentLength((int) file.length());
response.setHeader("Content-Disposition", "inline;filename=\"" + fileName + "\"");
FileCopyUtils.copy(inputStream, response.getOutputStream());
}
}