Below is the piece of code that causes OutOfMemory issue when I run my xyz.war in tomcat 7 with Java 8.
In Below code I am creating a CSV response of the data that was fetched from MongoDB via cursor.
#RequestMapping(value = "/elements/{elementname}/records", method = RequestMethod.GET)
public ModelAndView getAllRecords(
HttpServletRequest request, HttpServletResponse response,
#RequestParam(value = "customerid", required = true) long customerId,
#RequestParam(value = "userid", required = true) long userId,
#RequestParam(value = "query", required = false) String query,
throws Exception {
Map < String, Object > model = new HashMap < String, Object > ();
JsonObject records = elementService.searchRecords(query);
ModelAndViewData msvd = elementService.commonRestService
.getModelAndView("dataObject", "streamingView");
return new ModelAndView(msvd.getViewName(), handleCsvReportTypeRequest(records, customerId, userId));
}
public Map < String, Object > handleCsvReportTypeRequest(JsonObject records,
String elementName, long customerId, long userId) throws Exception {
StringBuffer csvData = new StringBuffer();
// create csv data
ModelAndViewData modelAndStreamingViewData = commonRestService.getModelAndView(
"dataObject", "streamingView");
byte[] byteArray = String.valueOf(csvData).getBytes();
InputStream byteArrayInputStream = new ByteArrayInputStream(byteArray);
model.put(modelAndStreamingViewData.getModelAttributeName(), byteArrayInputStream);
model.put(DownloadConstants.CONTENT_TYPE, DownloadConstants.CSV_CONTENT_TYPE);
model.put(DownloadConstants.FILENAME, "XYZ.csv");
model.put(DownloadConstants.LAST_MODIFIED, new Date(System.currentTimeMillis()));
model.put(DownloadConstants.CONTENT_LENGTH, Integer.valueOf(byteArray.length));
return model;
}
How can I stream CSV data back to the user without creating a huge data in memory and then passing to the user?
Use a Buffered read and write the response in HttpResponse object.
Try this way:
Spring MVC : large files for download, OutOfMemoryException
Related
Please Could someone help me , I cant figure out what is the problem, I'am trying to implement a test to this method but it always gives me
java.lang.AssertionError: Status
Expected :200
Actual :400
#PutMapping("/infoUtile/update/{id}")
public Map<String,Object> editActualite(#PathVariable Long id, #Valid #RequestParam ArrayList<Long> idDeleted,
#Valid #RequestParam String content, #Valid #RequestParam String description){
InformationUtile info = this.infoUtileService.getInfoUtileById(id);
info.setContent(content);
info.setDescription(description);
info.setDate(new Date());
if(idDeleted.size() != 0) {
for (int i = 0; i < idDeleted.size(); i++) {
this.mediaService.deleteMedia(idDeleted.get(i));
}
}
InformationUtile i = this.infoUtileService.addOrEditInfoUtile(info);
return getInfoUtileWeb(i);
}
and here is my test that Im trying to implement
#Test
public void update() throws Exception {
InformationUtile informationUtile = new InformationUtile();
informationUtile.setId(1);
informationUtile.setContent("oumaima");
informationUtile.setDescription("test");
Media medias = new Media();
medias.setId(1);
medias.setType("image/png");
medias.setUrl("C:\\files\\actualite\\32769\\adobexd.png");
List<Media> allMedias = new ArrayList<Media>();
allMedias.add(medias);
informationUtile.setMedias(allMedias);
User user = new User();
user.setId(1);
user.setNom("oumaima");
informationUtile.setUser(user);
ArrayList<Long> idDeleted = new ArrayList<>();
idDeleted.add(0L);
Mockito.when(informationUtileService.getInfoUtileById(Mockito.<Long>any())).thenReturn(new InformationUtile());
Mockito.when(informationUtileService.addOrEditInfoUtile(Mockito.any(InformationUtile .class))).thenReturn(informationUtile);
mockMvc.perform(put("/infoUtile/update/{id}",informationUtile.getId()).requestAttr("idDeleted",idDeleted)
.param("content",informationUtile.getContent())
.param("description",informationUtile.getDescription())
)
.andExpect(status().isOk());
verify(informationUtileService, times(1)).getInfoUtileById(informationUtile.getId());
verify(informationUtileService, times(1)).addOrEditInfoUtile(informationUtile);
verifyNoMoreInteractions(informationUtileService);
}
You are defining three request parameters at your endpoint #Valid #RequestParam ArrayList<Long> idDeleted, #Valid #RequestParam String content, #Valid #RequestParam String description which means they are query parameters after the url, e.g. http://localhost:8080/?idDeleted=1&idDeleted=2&content=Hello&description=Duke.
The HTTP 404 indicates that Spring could not find a handler for your request, meaning the client (in your case MockMvc) has a malformed URL.
In your current MockMvc request setup you are using .requestAttr() for the idDeleted request parameter.
All of them should be .param():
mockMvc
.perform(put("/infoUtile/update/{id}",informationUtile.getId())
.param("idDeleted", idDeletedOne , idDeletedTwo)
.param("content",informationUtile.getContent())
.param("description",informationUtile.getDescription())
)
PS: I guess the #Valid annotations are redundant/not needed here as you are not checking e.g. payload which has Bean Validation annotations to verify the content.
UPDATE: .param() is overloaded with .parm(String name, String... values), so you can pass your list of idDeleted with either .param("idDeleted", idDeletedOne, idDeletedTwo) or you can pass a String[] with all your Long values represented as a String
These are two SpringBoot projects. A is for api service providing, and B is consuming A's service via rest template. It is OK while transferring string. While transferring Excel file via byte array, B receives a string, not byte array.
The api method in A.
#GetMapping(value = "/export")
#ResponseBody
public HSResult exportFile(#RequestParam(value = "fileName", defaultValue = "-1") String fileName,
#RequestParam(value = "provider", defaultValue = "-1") String channelId,
#RequestParam(value = "fileStatus", defaultValue = "-5") int fileStatus,
#RequestParam(value = "cdate", defaultValue = "") String cdate,
HttpServletRequest request, HttpServletResponse response) {
// ...... some logic code
InputStream inputStream=new FileInputStream(fullPath);
long fileSize=new File(fullPath).length();
byte[] allBytes = new byte[(int) fileSize];
inputStream.read(allBytes);
result = HSResult.build(200, "exportFile success",allBytes);
return result;
}
The consuming method in B.
public ResponseEntity downLoadFile(int businessType, String fileName, int fileStatus, HttpServletResponse response) {
//.......
ResponseEntity<HSResult> responseEntity = restTemplate.getForEntity(url, HSResult.class);
HSResult apiResult = responseEntity.getBody();
byte[] fileData = (byte[]) apiResult.getData();
//.......
}
A reads an excel file from disk into a byte array before transferring.
But while receving the result in B side, it is string like UEsDBC0AAAAIADhMuVAKVjbC//////////8LABQAX3Jl
Why did this happen? And how to transfer byte array through rest template correctly? Thanks.
PS: The class of HSResult
public class HSResult {
private Integer status;
private String msg;
private Object data;
//gets and setters
//......
}
Finally, I find the root cause. Share it with who may encounter the same issue.
In A side, it puts a byte array in the data field of HSResult, this field is in type object.
While receving this in B side, rest template is trying to cast all the data back into HSResult. When it comes to the byte array, the receving field data is in type object. So rest template does not konw what the specific type it is, then convert the byte array into a string, with some decode. I don't know whether it is exactly UTF8 or GB2312 or else.
How to resolve this? Just specify the receving field with specific type, not object.
public class HSResult {
private Integer status;
private String msg;
private byte[] data;
//gets and setters
}
Then everythiing is OK.
Thanks for Ken's reminder. Http is a text protocal, it cannot transfer byte array directly, so byte array must be converted into string at first.
I have used the Gson() class to convert object to json! and it has no problem with byte arrays.
I have this problem when I want to move my codes to spring, so I solved it by serializing byte[] filed:
class ByteArraySerializer: JsonSerializer<ByteArray>() {
override fun serialize(bytes: ByteArray, jsonGenerator: JsonGenerator, serializerProvider: SerializerProvider?) {
val intArray = intArray(bytes)
jsonGenerator.writeArray(intArray, 0, intArray.size)
}
private fun intArray(input: ByteArray): IntArray {
val ret = IntArray(input.size)
for (i in input.indices) ret[i] = input[i].toInt() // & 0xff -> 0-255
return ret
}
}
convert byte array to int array
and then use it in my class:
data class VideoInfo(val durationMS: Int = 0): Serializable {
#JsonSerialize(using = ByteArraySerializer::class)
var bytes: ByteArray? = null
}
It will return json object as a below:
{
"bytes": [-1,-40, ..., 0],
"durationMS": 8870
}
It is kotlin, You can easily convert it to java :)
I have a spring boot web application which will handle large file (max size of 5g) upload and then save it to s3. The request and response may last for a long time.
So what is the best practice to handle the upload and download like this? How to make a good performance to prevent my server down when download or upload large files?
you can use multipart/form-data
#RequestMapping(value = "/agency/create", method = RequestMethod.POST, consumes = "multipart/form-data")
public ResponseEntity<List<String>> createAgency(
#RequestParam(value = "username", required = true) String username,
#RequestParam(value = "pic1", required = true)MultipartFile pic1File,
MultipartHttpServletRequest request, ModelAndView modelAndView) {
List<String> requestKeys=new ArrayList<String>();
List<String> originalFileName=new ArrayList<String>();
request.getFileNames().forEachRemaining(requestKeys::add);
for(String multipartFile:requestKeys) {
originalFileName.add(request.getFile(multipartFile).getOriginalFilename());
}
storageService.store(pic1File);
return new ResponseEntity<List<String>>(originalFileName, HttpStatus.CREATED);
}
Posting in case someone finds this useful in the future. This works with a REST controller as of Spring Boot 2.4.2.
Class annotations:
#RestController
#RequestMapping("/api")
Method declaration:
#RequestMapping(path = "/file-upload/{depot}/{fileName}", method = {RequestMethod.POST, RequestMethod.PUT})
public ResponseEntity<String> fileUpload(
#PathVariable(name = "depot") String depot,
#PathVariable(name = "fileName") String fileName,
InputStream inputStream,
HttpServletRequest request,
HttpServletResponse response)
The above is the Spring Boot configuration for a REST Controller that worked for me for large file upload. The key was adding InputStream inputStream directly.
I would like to send a File object along with custom model object in a single request.
let formData:FormData = new FormData();
let file = this.fileList[0];
formData.append('file', file, file.name);
formData.append('address', JSON.stringify(customObj));
...
this.http.post(fileServeUrl, formData)
My backend is in Spring Rest as below
#RequestMapping(value = "/fileServe",
produces = {"application/json"},
consumes = {MediaType.APPLICATION_JSON_VALUE, MediaType.MULTIPART_FORM_DATA_VALUE},
method = RequestMethod.POST)
ResponseEntity<Image> uploadFile(#RequestPart("file") MultipartFile imageData, #RequestPart("address") Address address) throws IOException {...}
I was able to receive the data if I pass simple String along with File though.
formData.append('file', file, file.name);
formData.append('address', addressText);
Backend
#RequestMapping(value = "/fileServe",
produces = {"application/json"},
consumes = {MediaType.APPLICATION_JSON_VALUE, MediaType.MULTIPART_FORM_DATA_VALUE},
method = RequestMethod.POST)
ResponseEntity<Image> uploadFile(#RequestPart("file") MultipartFile imageData, #RequestPart("address") String addressText) throws IOException {...}
I tried #RequestBody for my custom object but even that didn't work. Any advise please.
The problem with #Requestbody and #RequestPart annotation is that spring use the HttpMessageConverter to take convert the incoming json message into the your object. As you send form data with a file and a text value spring can not convert it into your object. I am afraid you have to pass the value of address seperatetly.
#RequestMapping(value = "/fileupload", headers = ("content-type=multipart/*"), method = RequestMethod.POST)
public ResponseEntity<AjaxResponseBody> upload(#RequestParam("file") MultipartFile file, #RequestParam String name, #RequestParam String postCode) {
AjaxResponseBody result = new AjaxResponseBody();
HttpHeaders headers = new HttpHeaders();
if (!file.isEmpty()) {
try {
Address address = new Address();
address.setName(name);
result.setMsg("ok");
return new ResponseEntity<AjaxResponseBody>(result, headers, HttpStatus.OK);
} catch (Exception e) {
return new ResponseEntity<AjaxResponseBody>(HttpStatus.BAD_REQUEST);
}
} else {
return new ResponseEntity<AjaxResponseBody>(HttpStatus.BAD_REQUEST);
}
}
Expept if you find a way your client app send a file with MimeType of image/jpg and and an address of application/json which allow spring to parse the json and map to your Address object which i couldn't do it.
When I click on any link the content should be downloaded
But this is what I get.
MastercourseController.java
#RequestMapping(value = { ControllerUriConstant.download_file }, method = RequestMethod.GET)
#ResponseBody
public void downloadingAFileById(#RequestParam("id") String id, Model model, HttpServletRequest request)
throws TechnoShineException, IOException {
String filePath = "D:/dev/testFIle.txt";
long download = Long.parseLong(id);
byte[] b = masterCourseFileFormService.getAllDownloadable(download);
OutputStream outputStream = new FileOutputStream(filePath);
outputStream.write(b);
outputStream.close();
}
MasterCourseService
public byte[] getAllDownloadable(long id) throws TechnoShineException
{
return masterCourseFormUploadDao.getAllDownloadableFiles(id);
}
MasterCourseDao
public byte[] getAllDownloadableFiles(long id) throws TechnoShineException
{
return masterCourseFormUploadMapper.getAllDownloadable(id);
}
MasterCourseMapper
public byte[] getAllDownloadable(long id) throws TechnoShineException;
You are writing the data returned by getAllDownloadable(..) to a hard-coded file. Are you sure that is what you want? I think you want to write the content returned by getAllDownloadable(..) to be written into the response. That can be done by adding a method parameter of the type HttpServletResponse to your mapping and writing into the output stream returned by HttpServletResponse#getOutputStream() and flushing (not closing!) that stream at the end.
Furthermore you have to remove the #ResponseBody annotation as this is meant to be used if the value that is returned by the mapping method returns the data that should directly be sent to the client (i.e. when sending a JSON data object or a string) without passing it to the template engine. As you are not returning anything you can remove this annotation.
Furthermore you have to set the content type of your response by invoking HttpServletResponse#setContentType(contentType: String).
In your case, the invocation would be the following:
response.setContentType("text/plain");
You complete method would look like this:
#RequestMapping(
value = ControllerUriConstant.download_file,
method = RequestMethod.GET
)
public void downloadingAFileById(#RequestParam("id") String id, HttpServletResponse response)
throws TechnoShineException, IOException {
long download = Long.parseLong(id);
byte[] b = masterCourseFileFormService.getAllDownloadable(download);
response.getOutputStream().write(b);
response.getOutputStream().flush();
}