The project I'm working on needs to support large file uploads and know the time taken during their upload.
To handle the large files I'm using the streaming API of Apache FileUpload, this also allows me to measure the time taken for the complete stream to be saved.
The problem I'm having is that I cannot seem to be able to utilise MockMvc in an Integration Test on this controller. I know that the controller works as I've successfully uploaded files using postman.
Simplified Controller Code:
#PostMapping("/upload")
public String handleUpload(HttpServletRequest request) throws Exception {
ServletFileUpload upload = new ServletFileUpload();
FileItemIterator iterStream = upload.getItemIterator(request);
while (iterStream.hasNext()) {
FileItemStream item = iterStream.next();
String name = item.getFieldName();
InputStream stream = item.openStream();
if (!item.isFormField()) {
// Process the InputStream
} else {
String formFieldValue = Streams.asString(stream);
}
}
}
Simplified Test Code:
private fun uploadFile(tfr: TestFileContainer) {
val mockFile = MockMultipartFile("file", tfr.getData()) // .getData*() returns a ByteArray
val receiveFileRequest = MockMvcRequestBuilders.multipart("/upload")
.file(mockFile)
.contentType(MediaType.MULTIPART_FORM_DATA)
val result = mockMvc.perform(receiveFileRequest)
.andExpect(status().isCreated)
.andExpect(header().exists(LOCATION))
.andReturn(
}
This is the error I'm currently getting
org.apache.tomcat.util.http.fileupload.FileUploadException: the
request was rejected because no multipart boundary was found
Can anyone help?
The MockMultipartFile approach won't work as Spring does work behind the scenes and simply passes the file around.
Ended up using RestTemplate instead as it actually constructs requests.
Related
I have a Spring boot API and one of the endpoints allows users to upload video's. Now My controller basically takes the file as a MultiPart file and then I store it in a temp folder accessible to tomcat. Once I have it stored on Disk, I then push the video to an S3 bucket.
Now to me anyway, this seems to be less than optimal, Like if I wanted to have a 100 or a 1000 users upload at once it seems really non performant to write the files to disk first.
As a little background I'm storing it on disk with the intention that if there is a issue pushing to S3 I can retry
The below code might show what I'm doing better than the above:
public Video addVideo(#RequestParam("title") String title,
#RequestParam("Description") String Description,
#RequestParam(value = "file", required = true) MultipartFile file) {
this.amazonS3ClientService.uploadFileToS3Bucket(file, title, description));
}
Method for storing Video file:
String fileNameWithExtenstion = awsS3FileName + "." + FilenameUtils.getExtension(multipartFile.getOriginalFilename());
//creating the file in the server (temporarily)
File file = new File(tomcatTempDir + fileNameWithExtenstion);FileOutputStream fos = new FileOutputStream(file);
fos.write(multipartFile.getBytes());
fos.close();PutObjectRequest putObjectRequest = new PutObjectRequest(this.awsS3Bucket, awsS3BucketFolder + UnigueId + "/" + fileNameWithExtenstion, file);
if (enablePublicReadAccess) {
putObjectRequest.withCannedAcl(CannedAccessControlList.PublicRead);
}
// Upload a file as a new object with ContentType and title
specified.amazonS3.putObject(putObjectRequest);
//removing the file created in the server
file.delete();
So my question is....is there a better way in Tomcat to:
A) Take in a file via a controllerB) Push to S3
There is no other way to do it with multipart. The problem with multipart that to properly segement parts from the requst they need sometimes skipped or be repeatable. That is impossible within memory w/o having memory to explode. Therefore, Commons FileUpload caches them on disk after a certain threshold is reached.
Multipart requests are the worst way for that. I highly recommend to use either PUT or POST with content type application/octet-stream. You can take the bare request input stream and pass to HttpClient to stream to your backend server. I did this already 5 years ago and it works for gigabytes. I have posted the solution in the Apache HttpClient mailing list.
There is one possibility how this could work under specific conditions:
All parts are in the correct physical order you want to read
Your write to a backend is fast enough to sustain the read from the front
Consume the root part and then go over to the next physical one, process the request body lazily. JAX-WS RI (Metro) has a very nice handling of multipart requests for XOP/MTOM. Learn from that because you won't be able to make it any better.
Perhaps you can try to direct stream the input stream from your MultipartFile to S3.
Consider the following uploadFileToS3Bucket method:
public PutObjectResult uploadFileToS3Bucket(InputStream input, long size, String title, String description) {
// Indicate the length of the information to avoid the need to compute it by the AWS SDK
// See: https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/PutObjectRequest.html#PutObjectRequest-java.lang.String-java.lang.String-java.io.InputStream-com.amazonaws.services.s3.model.ObjectMetadata-
ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.setContentLength(size); // rely on Spring implementation. Maybe you probably also can use input.available()
// compute the object name as appropriate
String key = "...";
PutObjectRequest putObjectRequest = new PutObjectRequest(
this.awsS3Bucket, key, input, objectMetadata
);
// The rest of your code
if (enablePublicReadAccess) {
putObjectRequest.withCannedAcl(CannedAccessControlList.PublicRead);
}
// Upload a file as a new object with ContentType and title
return specified.amazonS3.putObject(putObjectRequest);
}
Of course, you need to provide the service the input stream obtained from the client request associated with the MutipartFile object:
public Video addVideo(
#RequestParam("title") String title,
#RequestParam("Description") String Description,
#RequestParam(value = "file", required = true) MultipartFile file) {
try (InputStream input = file.getInputStream()) {
this.amazonS3ClientService.uploadFileToS3Bucket(input, file.getSize(), title, description));
}
}
Probably you can also play with the getBytes method of MultipartFile and create a ByteArrayInputStream to perform the operation.
In addVideo:
byte[] bytes = file.getBytes();
In uploadFileToS3Bucket:
ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.setContentLength(bytes.length);
PutObjectRequest putObjectRequest = new PutObjectRequest(
this.awsS3Bucket, key, new ByteArrayInputStream(bytes), objectMetadata
);
I would prefer the first solution, but try to determine which option offers you the best performance.
I am struggling with uploading an image from thew client-side to a folder on the server-side in .Net Core.I used Postman to check if the method on the server-side is working and it does without any problem,but when I try to upload an image from the client-side,I get an error on the server-side of type NullReferenceException:Object reference not set to an instance of an object.This is the Post method on the server-side:
[HttpPost]
public async Task Post(IFormFile file)
{
if (string.IsNullOrWhiteSpace(_environment.WebRootPath))
{
_environment.WebRootPath = Path.Combine(Directory.GetCurrentDirectory(), "wwwroot");
}
var uploads = Path.Combine(_environment.WebRootPath, "uploads");
//var fileName = file.FileName.Split('\\').LastOrDefault().Split('/').LastOrDefault();
if (!Directory.Exists(uploads)) Directory.CreateDirectory(uploads);
if (file.Length > 0)
{
using (var fileStream = new FileStream(Path.Combine(uploads, file.FileName), FileMode.Create))
{
await file.CopyToAsync(fileStream);
}
}
}
Apparently the method is thrown where I check if the length of the file is bigger than 0.On the client-side I get error "500 internal server error" and I tried to check using the debugger where exactly the error is thrown but i can't find anything that could resemble an error of some sort.This is the API method for the client-side:
public async Task UploadPictureAsync(MediaFile image)
{
User user = new User();
string pictureUrl = "http://10.0.2.2:5000/api/UploadPicture";
HttpContent fileStreamContent = new StreamContent(image.GetStream());
// user.Picture=GetImageStreamAsBytes(image.GetStream());
fileStreamContent.Headers.ContentDisposition = new ContentDispositionHeaderValue("form-data") {FileName=Guid.NewGuid() + ".Png",Name="image"};
fileStreamContent.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
HttpClientHandler clientHandler = new HttpClientHandler();
clientHandler.ServerCertificateCustomValidationCallback = (sender, cert, chain, sslPolicyErrors) => { return true; };
using (var client = new HttpClient(clientHandler))
{
using (var formData = new MultipartFormDataContent())
{
formData.Add(fileStreamContent);
var response = await client.PostAsync(pictureUrl, formData);
if(response.IsSuccessStatusCode)
{
var result = response.Content.ReadAsStringAsync().Result;
}
}
}
}
The image is declared in the Model as byte array:
public byte[] Picture { get; set; }
Does someone understand why my POST method has this behavior since the server-side works perfectly but fails when I try to upload an image from the client-side?What I find weird though is that when i read the error and I look at the Content-Type it is "text/plain" instead of "form-data" and I have tried to set it at the MutipartFormDataContent like this:
formData.Headers.ContentType.MediaType = "multipart/form-data";
I also tried to set the MediaTypeHeaderValue on the client like this:
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/octet-stream"));
I still get the wrong content type.
I have also tried a different approach with Stream instead of MediaFile but without any luck as it did not even hit the break point in debugger mode for the response.Any help would be appreciated! :)
I have managed to find the answer finalllyyyyy!!!The problem was on the client-side as I suspected and guess what,it was all about the correct name.It turns out that since on the server side I have IFormFile file I had to change the client side to take the parameter name "file" instead of image as well so that it could work.Thank you #Jason for the suggestions as I didn't understand the error from the first place and did some debugging on the server-side to help me figure it out.
I have the following controller method:
#PostMapping(consumes = MediaType.MULTIPART_FORM_DATA_VALUE, path = "/upload")
public Mono<SomeResponse> saveEnhanced(#RequestPart("file") Mono<FilePart> file) {
return documentService.save(file);
}
which calls a service method where I try to use a WebClient to put some data in another application:
public Mono<SomeResponse> save(Mono<FilePart> file) {
MultipartBodyBuilder bodyBuilder = new MultipartBodyBuilder();
bodyBuilder.asyncPart("file", file, FilePart.class);
bodyBuilder.part("identifiers", "some static content");
return WebClient.create("some-url").put()
.uri("/remote-path")
.syncBody(bodyBuilder.build())
.retrieve()
.bodyToMono(SomeResponse.class);
}
but I get the error:
org.springframework.core.codec.CodecException: No suitable writer found for part: file
I tried all variants of the MultipartBodyBuilder (part, asyncpart, with or without headers) and I cannot get it to work.
Am I using it wrong, what am I missing?
Regards,
Alex
I found the solution after getting a reply from one of the contributes on the Spring Framework Github issues section.
For this to work:
The asyncPart method is expecting actual content, i.e. file.content(). I'll update it to unwrap the part content automatically.
bodyBuilder.asyncPart("file", file.content(), DataBuffer.class)
.headers(h -> {
h.setContentDispositionFormData("file", file.name());
h.setContentType(file.headers().getContentType());
});
If both headers are not set then the request will fail on the remote side, saying it cannot find the form part.
Good luck to anyone needing this!
Trying to upload an image and it keeps sending as just bytes, not an image file. This is a very simple call, I don't need to send any params other than the image itself. I don't know how to format logs so I won't post the error here unless requested to.
The service:
public interface FileUploadService {
#Multipart
#POST("upload_profile_picture")
Call<ResponseBody> uploadProfilePicture(#Part("profile_picture") RequestBody file);
}
The call being made (a file is generated earlier, had to remove this code because SO needs the post to be mainly words..dumb..):
// Generate the service from interface
FileUploadService service = ServiceGenerator.createService(FileUploadService.class, this);
// Create RequestBody instance from file
RequestBody requestFile =
RequestBody.create(MediaType.parse("image/*"), imageFile);
Log.d(LOG_TAG, "formed file");
// finally, execute the request
Call<ResponseBody> call = service.uploadProfilePicture(requestFile);
Log.d(LOG_TAG, "sending call");
call.enqueue(new Callback<ResponseBody>() {
#Override
public void onResponse(Call<ResponseBody> call,
Response<ResponseBody> response) {
Log.d(LOG_TAG, "success");
Log.d(LOG_TAG, response.toString());
}
#Override
public void onFailure(Call<ResponseBody> call, Throwable t) {
Log.d(LOG_TAG, "failure");
Log.e(LOG_TAG, t.getMessage());
}
});
Is the issue with the MediaType.parse method? I've tried "multipart/form-data", "image/jpeg", and the above as well and nothing has worked.
The server team has said they are receiving the call, just as bytes and no image file.
I keep getting a 400 because it's sending all bytes. How can I just send this? Do I need to send as a multipart or what? From what I've seen, you just need to tag the param in the method with #Body and do the above and it should all work. Can anybody tell me why this is happening? Thanks!
This is a known issue in Retrofit 2.
Edit: Support for OkHttp's MultipartBody.Part has been added in the final 2.0 release.
In order to get it working, you need to change your interface a little bit first:
#Multipart
#POST("upload_profile_picture")
Call<ResponseBody> uploadProfilePicture(#Part MultipartBody.Part file);
Then you have to create the Part and make the call like this:
MultipartBody.Part file = MultipartBody.Part.createFormData(
"file",
imageFile.getName(),
RequestBody.create(MediaType.parse("image/*"), imageFile));
Call<ResponseBody> call = service.uploadProfilePicture(file);
I am using Jersey JAX-RS.
I want to return a Response with a Map containing Image Files and JSON String values.
Is this the right way to do this:
Map<String,Object> map = new HashMap........
GenericEntity entity = new GenericEntity<Map<String,Object>>(map) {};
return Response.ok(entity).build();
Or is this better.I plan to use JAX-RS with Jersey only.
JResponse.ok(map).build();
I am basing this on this article:
http://aruld.info/handling-generified-collections-in-jersey-jax-rs/
I am not sure what to specify for #Produces too(planning to leave it out).
TIA,
Vijay
You better produce a multipart response:
import static com.sun.jersey.multipart.MultiPartMediaTypes.MULTIPART_MIXED_TYPE;
import static javax.ws.rs.core.MediaType.APPLICATION_XML_TYPE
#GET
#Produces(MULTIPART_MIXED_TYPE)
public Response get()
{
FileDataSource image = ... (gets the image file)
String info = ... (gets the xml structured information)
MultiPart multiPart = new MultiPart().
bodyPart(new BodyPart(info, APPLICATION_XML_TYPE)).
bodyPart(new BodyPart(image, new MediaType("image", "png")));
return Response.ok(multiPart, MULTIPART_MIXED_TYPE).build();
}
This example was taken from there.