I have the following controller:
#RequestMapping(value = "/url", method = RequestMethod.GET)
public List<ResponseEntity<InputStreamResource>> getLayouts(#RequestParam("app") String app, HttpServletResponse response) {
.....
inputStreamResource = new InputStreamResource(new FileInputStream(file));
HttpHeaders responseHeaders = new HttpHeaders();
responseHeaders.set(HttpHeaders.CONTENT_TYPE, "application/xml");
responseHeaders.setContentLength(file.length());
responseHeaders.add(HttpHeaders.CONTENT_ENCODING, "UTF-8");
ResponseEntity inputStreams = new ResponseEntity(inputStreamResource, responseHeaders, HttpStatus.OK);
inputStreamResources.add(inputStreams);
.....
}
So the problem is that it doesn't work. First I had the problem:
No serializer found for class java.io.FileDescriptor
and no properties discovered to create BeanSerializer
(to avoid exception, disable SerializationFeature.FAIL_ON_EMPTY_BEANS) )
Then I added:
objectMapper.setVisibility(PropertyAccessor.ALL, Visibility.NONE);
objectMapper.setVisibility(PropertyAccessor.FIELD, Visibility.ANY);
And problem disappeared, but arise the new one: I see that my inputStream repeated many times and now I have an error:
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:693) ~[jackson-databind-2.6.5.jar:2.6.5]
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:675) ~[jackson-databind-2.6.5.jar:2.6.5]
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:157) ~[jackson-databind-2.6.5.jar:2.6.5]
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:693) ~[jackson-databind-2.6.5.jar:2.6.5]
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:675) ~[jackson-databind-2.6.5.jar:2.6.5]
My question what's wrong I do or missed something? Is there better way to send list of files, maybe my approach is wrong in general?
Depends what you want to do, if you want to return paths and content then I would return something like ResponseEntity<List<FileResource>> where FileResource would be your own class:
public class FileResource {
private String fileName;
private byte[] content;
// getters, setters
}
But be aware though that this approach may consume a lot of memory and also result in huge responses - for larger files you may want to return only a list of file names and then let the client request the files one by one, whereupon you'd stream the file content directly to output. Or you could zip the files in a single archive and return the archive to the client (like many cloud storages do).
You may also want to look at https://spring.io/guides/gs/uploading-files/
Related
I'm trying to use Spring WebClient to make some basic REST API calls. I'm getting an error that the request is malformed, but I can't tell exactly why. Is there any way to easily log the contents of the request (really, just the request body)? Everything I find online is super complicated. Here's what I have:
LinkedMultiValueMap params = new LinkedMultiValueMap();
params.add("app_id", getOneSignalAppId());
params.add("included_segments", inSegment);
params.add("content_available", true);
params.add("contents", new LinkedMultiValueMap() {{
add("en", inTitle);
}});
BodyInserters.MultipartInserter inserter = BodyInserters.fromMultipartData(params);
WebClient client = WebClient.builder()
.baseUrl("https://onesignal.com")
.defaultHeader(HttpHeaders.AUTHORIZATION, "Basic " + getOneSignalKey())
.defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
.defaultHeader(HttpHeaders.ACCEPT, MediaType.APPLICATION_JSON_VALUE)
.build();
Mono<NotificationResponse> result = client
.post()
.uri("/api/v1/notifications")
.body(inserter)
.retrieve()
.bodyToMono(NotificationResponse.class);
I just want a string of the JSON that will be inserted into the request body.
You can create your own wrapper/proxy class around the JSON encoder (assuming you're using JSON) and intercept the serialized body before it is sent into the intertubes.
If your request is going to send JSON.
Specifically, you would extend the encodeValue method (or encodeValues in case of streaming data) of Jackson2JsonEncoder (the default encoder). Then you can do with that data what you wish, such as logging etc. And you could even do this conditionally based on environment/profile.
This custom logging-encoder can be specified when creating the WebClient, by providing it as a codec:
CustomBodyLoggingEncoder bodyLoggingEncoder = new CustomBodyLoggingEncoder();
WebClient.builder()
.codecs(clientDefaultCodecsConfigurer -> {
clientDefaultCodecsConfigurer.defaultCodecs().jackson2JsonEncoder(bodyLoggingEncoder);
clientDefaultCodecsConfigurer.defaultCodecs().jackson2JsonDecoder(new Jackson2JsonDecoder(new ObjectMapper(), MediaType.APPLICATION_JSON));
})
...
I made a blog post about this. You might be able to find the encoder for Multipart data and apply similar principles
For completeness, the encoder might look something like this:
public class CustomBodyLoggingEncoder extends Jackson2JsonEncoder {
#Override
public DataBuffer encodeValue(final Object value, final DataBufferFactory bufferFactory,
final ResolvableType valueType, #Nullable final MimeType mimeType, #Nullable final Map<String, Object> hints) {
// Encode/Serialize data to JSON
final DataBuffer data = super.encodeValue(value, bufferFactory, valueType, mimeType, hints);
// This is your code:
SomethingAmazing.doItWithThisData(extractBytes(data));
// Return the data as normal
return data;
}
private byte[] extractBytes(final DataBuffer data) {
final byte[] bytes = new byte[data.readableByteCount()];
data.read(bytes);
// We've copied the data above to our array, but must reset the buffer for actual usage
data.readPosition(0);
return bytes;
}
}
Hope that helps somehow!
I am searching for a way to test a method, which sends a POST request to an external service. The application will NOT be itself a consumable webservice, that is why I didn't implement the shown class below as #RestController, #Controller, #Service, whatever types there may be.
But I don't know how to call the method postNumberPlate() to send a request to an embedded webserver (started in/by/at the unit test) to make some assertions on it. I want to avoid, to install an external webserver.
In other words: Can I start an embedded webserver inside a unit-test and 'tell' it to accept my POST request to inspect and assert the contents?
I already did:
a massive Webresearch (2-3 days?)
read Howto's
check the springboot docs
use an embedded Jetty Server (somehow blocking loop)
declare the Application as Webapplication and setting random port to jetty
experiment with Mockito, MockMVC
read "How to unittest a class using RestTemplate offline?" and compared it to my case, but found,
that it's very old (8y),
I don't know how to implement the parent interface, which is pretty huge
that the question and answers are too generic to deduce a solution for my case
it's not answering the embedded testing webserver problem I included.
The Class to be tested:
public class RestfulClient {
private RestTemplate restTemplate = new RestTemplate();
private HttpHeaders headers = new HttpHeaders();
#Value("${kafkaeskadapter.uri}")
private String destinationURL;
public RestfulClient() {}
public ResponseEntity<String> postNumberPlate(String key, CamImage camImage) {
LinkedMultiValueMap<String, Object> map = new LinkedMultiValueMap<>();
map.add("numplate", camImage.getIdentifier());
headers.setContentType(MediaType.MULTIPART_FORM_DATA);
HttpEntity<LinkedMultiValueMap<String, Object>> requestEntity = new HttpEntity<LinkedMultiValueMap<String,
Object>>(map, headers);
ByteArrayResource resource = new ByteArrayResource(camImage.getData()) {
/**
* IMPORTANT!!! Otherwise I receive a BAD REQUEST
* #return
*/
#Override
public String getFilename() {
return camImage.getIdentifier() + ".png";
}
};
map.add("image", resource);
ResponseEntity<String> result = restTemplate.exchange(destinationURL, HttpMethod.POST,
requestEntity, String.class);
return result;
}
}
I hope I could clarify my question a bit.
A solution is to write a simple light-weight Webservice Endpoint and include it into your Run Configuration of your IDE. I made a separate mini project and would add further methods if needed, e.g. to accept different media.
Prior to run the actual unit tests, it is possible to configure the start of the Endpoint and return a meaningful ResponseEntity. The result can be inspected et voilà, assertions are possible.
A word about StackOverflow user arrogance: #Raedwald, after reading and trying, the answers in the linked question are not really helpful, but involve a lot of knowlegde about the stuff, and I have no one around of my colleagues, which could ever assist at programming. So it wasn't helpful to flag my question for deletion.
I have a RestController that multiple partners use to send XML requests. However this is a legacy system that it was passed on to me and the original implementation was done in a very loose way in PHP.
This has allowed to clients, that now they refuse to change, to send different content-types (application/xml, text/xml, application/x-www-form-urlencoded) and it has left me with the need to support many MediaTypes to avoid returning 415 MediaType Not Supported Errors.
I have used the following code in a configuration class to allow many media types.
#Bean
public MarshallingHttpMessageConverter marshallingMessageConverter() {
MarshallingHttpMessageConverter converter = new MarshallingHttpMessageConverter();
converter.setMarshaller(jaxbMarshaller());
converter.setUnmarshaller(jaxbMarshaller());
converter.setSupportedMediaTypes(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.APPLICATION_XML,
MediaType.TEXT_XML, MediaType.TEXT_PLAIN, MediaType.APPLICATION_FORM_URLENCODED, MediaType.ALL));
return converter;
}
#Bean
public Jaxb2Marshaller jaxbMarshaller() {
Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
marshaller.setClassesToBeBound(CouponIssuedStatusDTO.class, CouponIssuedFailedDTO.class,
CouponIssuedSuccessDTO.class, RedemptionSuccessResultDTO.class, RedemptionResultHeaderDTO.class,
RedemptionFailResultDTO.class, RedemptionResultBodyDTO.class, RedemptionDTO.class, Param.class,
ChannelDTO.class, RedeemRequest.class);
Map<String, Object> props = new HashMap<>();
props.put(javax.xml.bind.Marshaller.JAXB_FORMATTED_OUTPUT, true);
marshaller.setMarshallerProperties(props);
return marshaller;
}
The controller method is this:
#PostMapping(value = "/request", produces = { "application/xml;charset=UTF-8" }, consumes = MediaType.ALL_VALUE)
public ResponseEntity<RedemptionResultDTO> request(
#RequestHeader(name = "Content-Type", required = false) String contentType,
#RequestBody String redeemRequest) {
return requestCustom(contentType, redeemRequest);
}
This endpoint is hit by all clients. It is only one last client giving me trouble. They are sending content-type = application/x-www-form-urlencoded; charset=65001 (UTF-8)": 65001 (UTF-8)
Due to the way the charset is sent, Spring Boot refuses to return anything but 415. Not even MediaType.ALL seems to have any effect.
Is there a way to make Spring allow this to reach me ignoring the content-type? Creating a filter and changing the content type was not feasible since the HttpServletRequest is not allowing to mutate the content-type. I am out of ideas but I really think there has to be a way to allow custom content-types.
UPDATE
If I remove the #RequestBody then I don't get the error 415 but I have no way to get the request body since the HttpServletRequest reaches the Controller action empty.
You best case is to remove the consumes argument from the RequestMapping constructor. The moment you have it added, spring will try to parse it into known type MediaType.parseMediaType(request.getContentType()) & which tries to create a new MimeType(type, subtype, parameters) and thus throws exception due to invalid charset format being passed.
However, if you remove the consumes, and you wanna validate/restrict the incoming Content-Type to certain type, you can inject HttpServletRequest in your method as parameter, and then check the value of request.getHeader(HttpHeaders.CONTENT_TYPE).
You also have to remove the #RequestBody annotation so Spring doesn't attempt to parse the content-type in attempt to unmarshall the body. If you directly attempt to read the request.getInputStream() or request.getReader() here, you will see null as the stream has already been read by Spring. So to get access to input content, use spring's ContentCachingRequestWrapper inject using Filter and then you can later repeatedly read the content as it's cached & not reading from original stream.
I am including some code snippet here for reference, however to see executable example, you can refer my github repo. Its a spring-boot project with maven, once you launch it, you can send your post request to http://localhost:3007/badmedia & it will reflect you back in response request content-type & body. Hope this helps.
#RestController
public class BadMediaController {
#PostMapping("/badmedia")
#ResponseBody
public Object reflect(HttpServletRequest request) throws IOException {
ObjectMapper mapper = new ObjectMapper();
JsonNode rootNode = mapper.createObjectNode();
((ObjectNode) rootNode).put("contentType", request.getHeader(HttpHeaders.CONTENT_TYPE));
String body = new String(((ContentCachingRequestWrapper) request).getContentAsByteArray(), StandardCharsets.UTF_8);
body = URLDecoder.decode(body, StandardCharsets.UTF_8.name());
((ObjectNode) rootNode).put("body", body);
return mapper.writerWithDefaultPrettyPrinter().writeValueAsString(rootNode);
}
}
#Component
public class CacheRequestFilter extends GenericFilterBean {
#Override
public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain chain)
throws IOException, ServletException {
HttpServletRequest cachedRequest
= new ContentCachingRequestWrapper((HttpServletRequest) servletRequest);
//invoke caching
cachedRequest.getParameterMap();
chain.doFilter(cachedRequest, servletResponse);
}
}
I have a spring boot web application which will handle large file (max size of 5g) upload and then save it to s3. The request and response may last for a long time.
So what is the best practice to handle the upload and download like this? How to make a good performance to prevent my server down when download or upload large files?
you can use multipart/form-data
#RequestMapping(value = "/agency/create", method = RequestMethod.POST, consumes = "multipart/form-data")
public ResponseEntity<List<String>> createAgency(
#RequestParam(value = "username", required = true) String username,
#RequestParam(value = "pic1", required = true)MultipartFile pic1File,
MultipartHttpServletRequest request, ModelAndView modelAndView) {
List<String> requestKeys=new ArrayList<String>();
List<String> originalFileName=new ArrayList<String>();
request.getFileNames().forEachRemaining(requestKeys::add);
for(String multipartFile:requestKeys) {
originalFileName.add(request.getFile(multipartFile).getOriginalFilename());
}
storageService.store(pic1File);
return new ResponseEntity<List<String>>(originalFileName, HttpStatus.CREATED);
}
Posting in case someone finds this useful in the future. This works with a REST controller as of Spring Boot 2.4.2.
Class annotations:
#RestController
#RequestMapping("/api")
Method declaration:
#RequestMapping(path = "/file-upload/{depot}/{fileName}", method = {RequestMethod.POST, RequestMethod.PUT})
public ResponseEntity<String> fileUpload(
#PathVariable(name = "depot") String depot,
#PathVariable(name = "fileName") String fileName,
InputStream inputStream,
HttpServletRequest request,
HttpServletResponse response)
The above is the Spring Boot configuration for a REST Controller that worked for me for large file upload. The key was adding InputStream inputStream directly.
I have created a Spring Boot 2 demo application with the Spring Initializr and added the controller below:
#Controller
#RequestMapping("/demo")
public class UploadController {
private final static Logger LOG = LoggerFactory.getLogger(UploadController.class);
#PostMapping("/upload")
public ResponseEntity<String> uploadFile(
#RequestParam("metadata") MultipartFile metadata,
#RequestParam("payload") MultipartFile payload) throws IOException {
ObjectMapper mapper = new ObjectMapper();
Map metadataMap = mapper.readValue(metadata.getInputStream(), Map.class);
LOG.info("Received call to upload file {}", metadataMap.get("filename"));
LOG.info("File size: {}", payload.getBytes().length);
LOG.info("File {} successfully uploaded", metadataMap.get("filename"));
return ResponseEntity.ok().build();
}
}
I then added an application.yaml file containing this configuration:
spring:
servlet:
multipart:
max-file-size: 2000000MB
max-request-size: 2000000MB
resolve-lazily: true
My goal is to have the controller parse and log the metadata file before it starts reading the payload file, but the resolve-lazily setting seems to be ignored by Boot: the code inside the controller won't be executed until the whole body is read.
I use the command below to test the controller:
curl -F metadata=#metadata.json -F payload=#payload.bin http://localhost:8080/demo/upload
Is there anything wrong with my code/configuration? Am I getting the meaning of the setting right?
At present, if you want to avoid reading (and buffering) the whole body all at once, I think you will have to provide your own parser, as described in the answers here. What would be really interesting (but generally unnecessary) would be to do so in the form of a new MultipartResolver implementation.
There are two existing implementations documented for interface MultipartResolver, and both supply a function setResolveLazily(boolean) (standard), (commons). I have tried with both, and neither seem to allow for parsing or streaming multipart files or parameters independently.
Default is "false", resolving the multipart elements immediately, throwing corresponding exceptions at the time of the resolveMultipart(javax.servlet.http.HttpServletRequest) call. Switch this to "true" for lazy multipart parsing, throwing parse exceptions once the application attempts to obtain multipart files or parameters.
Despite what it says in the documentation, I have found that once you call resolveMultipart, the entire body is parsed and buffered before the call returns. I know this because I can watch the temp-files being created.
One note about "Is there anything wrong with my code"...
Answer: Yes, because by using #RequestParam you have indirectly asked Spring to resolve your parameters ahead of time, before your controller is ever called. What you should be able to do instead (if the documentation were correct) is request the parameters independently from inside your controller:
Configuration (application.properties):
spring.servlet.multipart.enabled = true
spring.servlet.multipart.resolve-lazily = true
Controller:
#PostMapping(path = "/upload", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
public ResponseEntity<Void> postUpload(HttpServletRequest rawRequest) {
multipartResolver.setResolveLazily(true); // unclear why this is exists
MultipartHttpServletRequest request = multipartResolver.resolveMultipart(rawRequest);
String p1 = request.getParameter("first-parameter");
String p2 = request.getParameter("second-parameter");
System.out.println("first-parameter="+p1+", second-parameter"+p2);
multipartResolver.cleanupMultipart(request);
return new ResponseEntity<Void>(HttpStatus.ACCEPTED);
}
One useful aspect of resolve-lazily that I have discovered is that it allows you to write your own parser for some rest controllers while using the built-in parser for others (see my answer here). In other words, you don't have to use spring.servlet.multipart.enabled = false to get your parser to work. This is a minor breakthrough relative to other advice that I had seen previously.