Cannot get Spring Boot to lazily resolve a multipart file - spring-boot

I have created a Spring Boot 2 demo application with the Spring Initializr and added the controller below:
#Controller
#RequestMapping("/demo")
public class UploadController {
private final static Logger LOG = LoggerFactory.getLogger(UploadController.class);
#PostMapping("/upload")
public ResponseEntity<String> uploadFile(
#RequestParam("metadata") MultipartFile metadata,
#RequestParam("payload") MultipartFile payload) throws IOException {
ObjectMapper mapper = new ObjectMapper();
Map metadataMap = mapper.readValue(metadata.getInputStream(), Map.class);
LOG.info("Received call to upload file {}", metadataMap.get("filename"));
LOG.info("File size: {}", payload.getBytes().length);
LOG.info("File {} successfully uploaded", metadataMap.get("filename"));
return ResponseEntity.ok().build();
}
}
I then added an application.yaml file containing this configuration:
spring:
servlet:
multipart:
max-file-size: 2000000MB
max-request-size: 2000000MB
resolve-lazily: true
My goal is to have the controller parse and log the metadata file before it starts reading the payload file, but the resolve-lazily setting seems to be ignored by Boot: the code inside the controller won't be executed until the whole body is read.
I use the command below to test the controller:
curl -F metadata=#metadata.json -F payload=#payload.bin http://localhost:8080/demo/upload
Is there anything wrong with my code/configuration? Am I getting the meaning of the setting right?

At present, if you want to avoid reading (and buffering) the whole body all at once, I think you will have to provide your own parser, as described in the answers here. What would be really interesting (but generally unnecessary) would be to do so in the form of a new MultipartResolver implementation.
There are two existing implementations documented for interface MultipartResolver, and both supply a function setResolveLazily(boolean) (standard), (commons). I have tried with both, and neither seem to allow for parsing or streaming multipart files or parameters independently.
Default is "false", resolving the multipart elements immediately, throwing corresponding exceptions at the time of the resolveMultipart(javax.servlet.http.HttpServletRequest) call. Switch this to "true" for lazy multipart parsing, throwing parse exceptions once the application attempts to obtain multipart files or parameters.
Despite what it says in the documentation, I have found that once you call resolveMultipart, the entire body is parsed and buffered before the call returns. I know this because I can watch the temp-files being created.
One note about "Is there anything wrong with my code"...
Answer: Yes, because by using #RequestParam you have indirectly asked Spring to resolve your parameters ahead of time, before your controller is ever called. What you should be able to do instead (if the documentation were correct) is request the parameters independently from inside your controller:
Configuration (application.properties):
spring.servlet.multipart.enabled = true
spring.servlet.multipart.resolve-lazily = true
Controller:
#PostMapping(path = "/upload", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
public ResponseEntity<Void> postUpload(HttpServletRequest rawRequest) {
multipartResolver.setResolveLazily(true); // unclear why this is exists
MultipartHttpServletRequest request = multipartResolver.resolveMultipart(rawRequest);
String p1 = request.getParameter("first-parameter");
String p2 = request.getParameter("second-parameter");
System.out.println("first-parameter="+p1+", second-parameter"+p2);
multipartResolver.cleanupMultipart(request);
return new ResponseEntity<Void>(HttpStatus.ACCEPTED);
}
One useful aspect of resolve-lazily that I have discovered is that it allows you to write your own parser for some rest controllers while using the built-in parser for others (see my answer here). In other words, you don't have to use spring.servlet.multipart.enabled = false to get your parser to work. This is a minor breakthrough relative to other advice that I had seen previously.

Related

Spring Boot: CRLF - Securely log payload in REST API

I have a Spring Boot app which exposes a REST API. I need to log the payload to be able to find errors in the JSON in the API calls.
I have ran a code analysis tools that reports the following security risk when I log the payload.
https://find-sec-bugs.github.io/bugs.htm#CRLF_INJECTION_LOGS
How can I protect against code injection? I guess removing new lines only protect against fake log entries and will not protect against code injection?
REST API:
#PostMapping("/my/api")
public ResponseEntity<String> handleApi(#RequestBody Body body) {
Payload logging:
#Slf4j
public class CustomRequestLoggingFilter extends AbstractRequestLoggingFilter {
private static final int MAX_PAYLOAD_LENGTH = 64000;
public CustomRequestLoggingFilter() {
this.setIncludeQueryString(true);
this.setIncludePayload(true);
this.setMaxPayloadLength(MAX_PAYLOAD_LENGTH);
}
#Override
public void afterRequest(HttpServletRequest request, String message) {
if (request.getRequestURI().equals("/my/api")) {
log.info(message); //This is the security risk
}
}
You can try to use OWASP Json Sanitizer library (https://owasp.org/www-project-json-sanitizer/migrated_content) to clean and sanitize Json input prior logging it. If you are not concerned about adding additional 3rd party dependency to your project.
NOTE: Last release of this library was in Jan 11, 2021
Example:
#Override
public void afterRequest(HttpServletRequest request, String message) {
if (request.getRequestURI().equals("/my/api")) {
String sanitizedJson = JsonSanitizer.sanitize(message);
log.info(sanitizedJson );
}
}
The linked report is suggesting a possible solution of replacing newlines to remove the risk:
log.info(message.replaceAll("[\r\n]",""));
You can manually sanitize each parameter.
log.info("User " + val.replaceAll("[\r\n]","") + " (" + userAgent.replaceAll("[\r\n]","") + ") was not authenticated");
Or using other solutions which change your logging configuration:
You can also configure your logger service to replace new line for all message events. Here is sample configuration for LogBack using the replace function.
<pattern>%-5level - %replace(%msg){'[\r\n]', ''}%n</pattern>
Finally, you can use a logger implementation that replace new line by spaces. The project OWASP Security Logging has an implementation for Logback and Log4j.
The vulnerability you are mentioning have nothing to do with code injection, only with the possibility of manipulating your logs.
Remediation for that, if you are producing plain text logs, is to sanitize that message (best with the OWASP library that Dmitriy suggested), but if you are managing your logs with some tool (e.g. ELK), probably you should produce logs in JSON format and that would automatically mitigate this issue for you.
Back to the code injection, considering you have set some max payload length, I don't think you can have any code injection in that log statement.

How do I unit test a Spring RestTemplate that takes a ResponseExtractor and RequestCallback?

I am developing in Groovy and I am trying to write a Spock unit test for the following use of Spring's RestTemplate...
Included are my request callback and response extractors, and my initialization class of the RestTemplate bean. I am using the ResponseExtractor to stream the response from GET myurl/ and copy it to a file. The RequestCallback is simply setting some headers on the request.
class RestTemplateConfig() {
#Bean(name = 'myRestTemplate')
RestTemplate getMyRestTemplate() {
RestTemplate restTemplate = new RestTemplateBuilder().build()
return restTemplate
}
}
class MyClass() {
#Autowired
#Qualifier('myRestTemplate')
RestTemplate restTemplate
File getFile() {
ResponseExtractor<Void> responseExtractor = { ClientHttpResponse response ->
// do something with the response
// in this case, the response is an input stream so we copy the input stream to a file
myFile = response.getBody() // roughly, in a psuedocode-ish way
return null
}
RequestCallback requestCallback = { ClientHttpRequest request ->
request.getHeaders().setAccept([MediaType.APPLICATION_JSON])
}
File myFile
// get my file data
restTemplate.execute('myurl/', HttpMethod.GET, requestCallback, responseExtractor)
return myFile
}
}
Spring framework docs for that particular execute(...) method: https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/web/client/RestTemplate.html#execute-java.net.URI-org.springframework.http.HttpMethod-org.springframework.web.client.RequestCallback-org.springframework.web.client.ResponseExtractor-
How do I mock out what's happening in these closures? Specifically, I'm interested in mocking out my response extractor because my current test always returns myFile as null.
when:
// do stuff
then:
1 * restTemplate.execute('myurl/, HttpMethod.GET, _, _) // how can I mock out the expected response here?
0 * _
myFile != null // this fails because myFile is null
After you updated your sample code as I requested, I can see more clearly now. You are suffering from a typical (non-)testability problem: Your method getFile does more than just getting a file. It instantiates two dependencies as local variables, making them unmockable and consequently the whole method mostly untestable.
So you want to refactor for better testability so as to be able to use one testing method I mentioned in my first comment:
If the requestCallback and responseExtractor can be injected via constructor or setter, you can inject mocks.
If they are created by some kind of factory class, you can stub that class.
In case of a factory method inside the class under test itself you can use a spy on the class and stub the factory method.
For a more general discussion of testability and how tests drive application design, see my other answer here, sections "General comments" and "Update".
If any of this is unclear, feel free to ask related(!) follow-up questions.

How to test a POSTing method by using an embedded Webserver in Springboot?

I am searching for a way to test a method, which sends a POST request to an external service. The application will NOT be itself a consumable webservice, that is why I didn't implement the shown class below as #RestController, #Controller, #Service, whatever types there may be.
But I don't know how to call the method postNumberPlate() to send a request to an embedded webserver (started in/by/at the unit test) to make some assertions on it. I want to avoid, to install an external webserver.
In other words: Can I start an embedded webserver inside a unit-test and 'tell' it to accept my POST request to inspect and assert the contents?
I already did:
a massive Webresearch (2-3 days?)
read Howto's
check the springboot docs
use an embedded Jetty Server (somehow blocking loop)
declare the Application as Webapplication and setting random port to jetty
experiment with Mockito, MockMVC
read "How to unittest a class using RestTemplate offline?" and compared it to my case, but found,
that it's very old (8y),
I don't know how to implement the parent interface, which is pretty huge
that the question and answers are too generic to deduce a solution for my case
it's not answering the embedded testing webserver problem I included.
The Class to be tested:
public class RestfulClient {
private RestTemplate restTemplate = new RestTemplate();
private HttpHeaders headers = new HttpHeaders();
#Value("${kafkaeskadapter.uri}")
private String destinationURL;
public RestfulClient() {}
public ResponseEntity<String> postNumberPlate(String key, CamImage camImage) {
LinkedMultiValueMap<String, Object> map = new LinkedMultiValueMap<>();
map.add("numplate", camImage.getIdentifier());
headers.setContentType(MediaType.MULTIPART_FORM_DATA);
HttpEntity<LinkedMultiValueMap<String, Object>> requestEntity = new HttpEntity<LinkedMultiValueMap<String,
Object>>(map, headers);
ByteArrayResource resource = new ByteArrayResource(camImage.getData()) {
/**
* IMPORTANT!!! Otherwise I receive a BAD REQUEST
* #return
*/
#Override
public String getFilename() {
return camImage.getIdentifier() + ".png";
}
};
map.add("image", resource);
ResponseEntity<String> result = restTemplate.exchange(destinationURL, HttpMethod.POST,
requestEntity, String.class);
return result;
}
}
I hope I could clarify my question a bit.
A solution is to write a simple light-weight Webservice Endpoint and include it into your Run Configuration of your IDE. I made a separate mini project and would add further methods if needed, e.g. to accept different media.
Prior to run the actual unit tests, it is possible to configure the start of the Endpoint and return a meaningful ResponseEntity. The result can be inspected et voilĂ , assertions are possible.
A word about StackOverflow user arrogance: #Raedwald, after reading and trying, the answers in the linked question are not really helpful, but involve a lot of knowlegde about the stuff, and I have no one around of my colleagues, which could ever assist at programming. So it wasn't helpful to flag my question for deletion.

How to redirect request to an URI passed in headers with Jetty AsyncProxyServlet

I'm creating a proxy micro-service with SpringBoot, Jetty and kotlin.
The purpose of this micro-service is to forward requests made by my front-end application to external services (avoiding CORS) and send back the response after checking some custom authentication. The query I'll receive will contain the URL of the target in the headers (i.e: Target-Url: http://domain.api/getmodel).
Based on this answer, I made a class that extends AsyncProxyServlet and overwrote the method sendProxyRequest :
class ProxyServlet : AsyncProxyServlet() {
private companion object {
const val TARGET_URL = "Target-Url"
}
override fun sendProxyRequest(clientRequest: HttpServletRequest, proxyResponse: HttpServletResponse, proxyRequest: Request) {
// authentication logic
val targetUrl = clientRequest.getHeader(TARGET_URL)
if (authSuccess) {
super.sendProxyRequest(clientRequest, proxyResponse, proxyRequest)
} else {
proxyResponse.status = HttpStatus.UNAUTHORIZED.value()
}
}
}
When I query my proxy, I get in this method and successfuly authenticate, but I fail to understand how to use my targetUrl to redirect the request.
The method keeps calling itself as it's redirecting the original request to itself (the request from http://myproxy:port/ to http://myproxy:port/).
It is very difficult to find documentation on this specific implementation of jetty, StackOverflow is my last resort!
First, setup logging for Jetty, and configure DEBUG level logging for the package namespace org.eclipse.jetty.proxy, this will help you understand the behavior much better.
The Request proxyRequest parameter represents a HttpClient/Request object, which is created with an immutable URI/URL destination (this is due to various other features that requires information from the URI/URL such as Connection pooling, Cookies, Authentication, etc), you cannot change the URI/URL on this object after the fact, you must create the HttpClient/Request object with the correct URI/URL.
Since all you want to do is change the target URL, you should instead be overriding the method ...
protected String rewriteTarget(HttpServletRequest clientRequest)
... and returning the new absolute URI String to the destination that you want to use (The "Target-Url" header in your scenario looks like a good candidate)
You can see this logic in the ProxyServlet.service(HttpServletRequest, HttpServletResponse) code block (which AsyncProxyServlet extends from)

How to enable Spring Reactive Web MVC to handle Multipart-file?

I'm trying to use the new reactive web-mvc implementation in a spring boot 2.0 application. I'm trying to define a method which consume multipart file but do not succeed at making it working :( - I always get a 415 error.
On one hand I have a controller containing the following request mapping :
#RequestMapping(method = RequestMethod.POST, path = "/myPath/{param}/{param2}", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
#ResponseBody
public Mono<Void> postFile(
#RequestBody MultipartFile data,
#PathVariable("param") String param,
#PathVariable("param2") String param2,
#RequestHeader(name = HEADER_DATE, required = false) #DateTimeFormat(pattern = DATE_FORMAT) Instant instant
){
return fileService.handleData(Mono.just(data), param, param2, instant);
}
On the other hand I had to add a server on the top of the basic dependencies as it seems netty do not handle multipart files. I so added the spring-boot-starter-tomcatdependency which enabled the MultipartAutoConfiguration to be matched and satisfied on application auto configuration.
When posting something using a curl call :
curl 'Meta-Date: 20170101104532' --form "file=#file.bin" http://localhost:8082/myPath/foo/bar
while debug logs are activated (logging.level.org.springframework.web=DEBUG) I got this exception :
org.springframework.web.server.UnsupportedMediaTypeStatusException: Request failure [status: 415, reason: "Content type 'multipart/form-data;boundary=------------------------58fa43b8f1a26de4' not supported"]
This error is thrown by the RequestBodyArgumentResolver which has the the following supported media types : [*/*, text/xml, application/*+json;charset=UTF-8, application/xml, text/plain;charset=UTF-8, application/x-www-form-urlencoded, application/json;charset=UTF-8] provided by 9 DecoderHttpMessageReader.
Before posting I also took a look at :
Spring MultiPart MediaType Unsupported which seems to not be relevant here as my autoconf report contains the following entry : MultipartAutoConfiguration#multipartResolver matched
set content-type to utf-8 with angularjs $http Adding a header setting Content-Transfer-Encoding: binary didn't changed anything.
My understanding is that Spring web 5.0 uses a new request decoder system as I don't find these classes on a spring 4 spring boot application, and there is not yet any DecoderHttpMessageReader dealing with multipart file
Did I miss something ? Or should I wait one to be implemented ?
Okay, It seems this is just not implemented for now as it currently exists a pull request for this feature : Add reactive multipart request support #1201
Should have check this earlier...
[EDIT] : The issue has been solved and merged into Spring master branch. Should no longer be an issue.
#PutMapping(value="/{..}",consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
public Mono<Void> save(#RequestPart("file") FilePart multipartFormData,#RequestParam("fileName") String fileName,#PathVariable("..") String ..) throws IOException {
List<ByteBuffer> bytesList = new LinkedList<>();
multipartFormData.content().
subscribe(item->bytesList.add(item.asByteBuffer()));
int totalBytes = bytesList.stream().mapToInt(item->item.capacity()).sum();
ByteBuffer buffer = ByteBuffer.allocate(totalBytes);
bytesList.stream().forEach(byteBuff->buffer.put(byteBuff));
baseImageHandler.saveImage(buffer, fileName, baseItemId);
return Mono.empty();
}
Please note that it is a dev verison, but this is how I have managed to do it.

Resources