I have this code:
#PostMapping("foobar")
public ResponseEntity<SaveLogsResult> foobar(#RequestPart("file") MultipartFile log, #RequestPart("env") MultipartFile json){
return ResponseEntity.ok(fooService.saveFooBar(log, json, UUID.randomUUID().toString()));
}
Two applications send formally correct data to this endpoint, one fails miserably and receives an http status 400.
I set logging.level.org.springframework.web=DEBUG and can see (among other lines) this:
Required request part 'env' is not present
Resolved [org.springframework.web.multipart.support.MissingServletRequestPartException: Required request part 'env' is not present]
Completed 400 BAD_REQUEST
To further diagnose this I compared a working (left) and a non-working (right) request. The only different is the mising filename:
As far as I understand the RFC for Content-Disposition leaving out the filename is perfectly valid:
Is followed by a string containing the original name of the file
transmitted. The filename is always optional and must not be used
blindly by the application: path information should be stripped, and
conversion to the server file system rules should be done. This
parameter provides mostly indicative information. When used in
combination with Content-Disposition: attachment, it is used as the
default filename for an eventual "Save As" dialog presented to the
user.
Is this an error inside Spring ? I use Spring Boot 2.6.2
Unfortunately I can't change the non-working component for a quick test because it is a bought component that doesn't receive bugfixes very often.
I think that my problem is different from that described here because the failure only happens in a specific scenario.
It looks like you have to provide the filename. see this issue
This [The situation in which filename is not present] can lead to inconsistencies, e.g. you would get it with
MultipartFile but not with collection or array of MultipartFile, and
one can make the opposite argument that it should be rejected.
Why does it matter to have it injected as MultipartFile if it doesn't
even have a filename? You could also inject it as InputStream or any
other type supported by a registered HttpMessageConverter.
Related
I have a question about the light-rest-4j URL validation, for example, if I have a POST request path /party, if I type the path as /party11, I will get error: No handler defined for path /party11, but if I put /party?qqqq, It will pass through, and system treat it as /party should we add validation for this? Our QA team creates this as a defect, in case user input it by mistake, they expect to have error message return.
The light-rest-4j framework validates the request/response based on the OpenAPI specification during the runtime; however, it only validates based on the spec — nothing more and nothing less. In most cases, the spec will define the type of headers, query parameters, path parameters, and cookies, as well as if they are required. We make sure these are validated as defined. For anything that is not defined in the spec, we are doing nothing. For example, an extra query parameter or an extra header in the request will be ignored as they are not defined in the spec. We cannot do any negative validation as we don't know if any client will add additional headers or query parameters for tracing, auditing, etc. A request that comes from one client might be different than another one comes from the same client through a gateway or proxy.
I have a Spring application that receives a request like http://localhost/foo?email=foo+bar#example.com. This triggers a controller that roughly looks like this:
#RestController
#RequestMapping("/foo")
public class FooController extends Controller {
#GetMapping
public void foo(#RequestParam("email") String email) {
System.out.println(email)
}
}
By the time I can access email, it's been converted to foo bar#example.com instead of the original foo+bar#example.com. According to When to encode space to plus (+) or %20? this should only happen in requests where the content is application/x-www-form-urlencoded. My request has a content type of application/json. The full MIME headers of the request look like this:
=== MimeHeaders ===
accept = application/json
content-type = application/json
user-agent = Dashman Configurator/0.0.0-dev
content-length = 0
host = localhost:8080
connection = keep-alive
Why is Spring then decoding the plus as a space? And if this is the way it should work, why isn't it encoding pluses as %2B when making requests?
I found this bug report about it: https://jira.spring.io/browse/SPR-6291 which may imply that this is fixed on version 3.0.5 and I'm using Spring > 5.0.0. It is possible that I may misinterpreting something about the bug report.
I also found this discussion about RestTemplate treatment of these values: https://jira.spring.io/browse/SPR-5516 (my client is using RestTemplate).
So, my questions are, why is Spring doing this? How can I disable it? Should I disable it or should I encode pluses on the client, even if the requests are json?
Just to clarify, I'm not using neither HTML nor JavaScript anywhere here. There's a Spring Rest Controller and the client is Spring's RestTemplate with UriTemplate or UriComponentsBuilder, neither of which encode the plus sign the way Spring decodes it.
Original Answer
You are mixing 2 things, a + in the body of the request would mean a space when header has application/x-www-form-urlencoded. The body or content of the request would be dependent on the headers but a request can just have a url and no headers and no body.
So the encoding of a URI cannot be controlled by any headers as such
See the URL Encoding section in https://en.wikipedia.org/wiki/Query_string
Some characters cannot be part of a URL (for example, the space) and some other characters have a special meaning in a URL: for example, the character # can be used to further specify a subsection (or fragment) of a document. In HTML forms, the character = is used to separate a name from a value. The URI generic syntax uses URL encoding to deal with this problem, while HTML forms make some additional substitutions rather than applying percent encoding for all such characters. SPACE is encoded as '+' or "%20".[10]
HTML 5 specifies the following transformation for submitting HTML forms with the "get" method to a web server.1 The following is a brief summary of the algorithm:
Characters that cannot be converted to the correct charset are replaced with HTML numeric character references[11]
SPACE is encoded as '+' or '%20'
Letters (A–Z and a–z), numbers (0–9) and the characters '*','-','.' and '_' are left as-is
All other characters are encoded as %HH hex representation with any non-ASCII characters first encoded as UTF-8 (or other specified encoding)
The octet corresponding to the tilde ("~") is permitted in query strings by RFC3986 but required to be percent-encoded in HTML forms to "%7E".
The encoding of SPACE as '+' and the selection of "as-is" characters distinguishes this encoding from RFC 3986.
And you can see the same behaviour on google.com as well from below screenshots
Also you can see the same behaviour in other frameworks as well. Below is an example of Python Flask
So what you are seeing is correct, you are just comparing it with a document which refers to the body content of a request and not the URL
Edit-1: 22nd May
After debugging it seems the decoding doesn't even happen in Spring. I happens in package org.apache.tomcat.util.buf; and the UDecoder class
/**
* URLDecode, will modify the source.
* #param mb The URL encoded bytes
* #param query <code>true</code> if this is a query string
* #throws IOException Invalid %xx URL encoding
*/
public void convert( ByteChunk mb, boolean query )
throws IOException
{
int start=mb.getOffset();
And below is where the conversion stuff actually happens
if( buff[ j ] == '+' && query) {
buff[idx]= (byte)' ' ;
} else if( buff[ j ] != '%' ) {
This means that it is an embedded tomcat server which does this translation and spring doesn't even participate in this. There is no config to change this behaviour as seen in the class code. So you have to live with it
SPR-6291 fixed this problem in v3.0.5 but this remains unresolved in some other cases like SPR-11047 is still unresolved. While SPR-6291's priority was Major, SPR-11047's priority is Minor.
I faced this problem when I was working on REST API in old Spring last year. There are multiple ways we can get data in Spring controller. So two of them are via #RequestParam or #PathVariable annotation
As others mentioned I think its spring's internal issue and does not specifically belong to URL encoding because I was sending data over POST request but it is somewhat encoding problem. But I also agree with others as now it remains problematic only in URL.
So there are two solutions I know:
You can use #PathVariable instead of #RequestParam because as of SPR-6291 this plus sign issue is fixed in #PathVariable and still remains open for #RequestParam as SPR-11047
My version of spring was not even accepting plus sign via #PathVariable annotation, so this is how I overcome the problem (I don't remember it step by step but it will give you hint).
In your case you can get the fields via JS and escape the plus sign before sending a request. Something like this:
var email = document.getElementById("emailField").value;
email = email.replace('+', '%2B');
If you have this request:
http://localhost/foo?email=foo+bar#example.com
then the original is foo bar#example.com. If you say the original should be foo+bar#example.com then the request should be:
http://localhost/foo?email=foo%2Bbar#example.com
So Spring is working as supposed to. Maybe on client you should check if the URI is properly encoded. The client-side URL encoding is responsible for building a correct HTTP request.
See encodeURI() if you generate the request in JavaScript or uriToString() if you generate the request in Spring.
Build your request string (the part after ?), without any encoding, with unencoded values like foo+bar#email.com, and only in the end, before actually using it in GET, encode all of it with whatever is available on the client platform. If you want to use POST then you should encode it according to the MIME type of your choice.
How to make Spring (or Jetty) accept/parse Multipart uploads where the body part(s) miss(es) the filename attribute in the Content-Disposition?
Otherwise than the missing filename attribute, the Multipart message is OK and it also used to work with an older version of Jetty/Spring.
What do I need to set to make Jetty/Spring a little more error tolerant again?
P.S.
Here I found a similar yet different problem (name attribute missing). Yet, while the name seems like a vital attribute to identify body parts, you do not necessarily need the original client-side filename: Spring POST multipart/form-data Request empty body, getParts always empty
As already mentioned in the comment:
Spring (4.3.8.RELEASE) unconditionally skips multipart files without a filename, i.e. there is no setting to make it more error-tolerant. Thus, the only solution was to override Spring's StandardMultipartHttpServletRequest class and modify the method in question: parseRequest.
The added lines of code:
if (filename == null && "file".equalsIgnoreCase(part.getName()) && MediaType.APPLICATION_OCTET_STREAM.equalsIgnoreCase(part.getContentType())) {
filename = DEFAULT_FILENAME;
}
And yes, the name of the multipart body part doesn't necessarily have to be "file" but this is the name the client application my code has to work with is using when uploading a file.
My task is to implement a webservice that:
consumes an XML file on a POST endpoint
in happy flow, it returns a DTO as JSON + HTTP 2xx
the incoming XML file is validated against a XSD; if the validation fails, a JSON with a list of all validation errors is returned (including the line, column, error) with HTTP Bad request
the application exposes two endpoints, only one of them should be validated
I have started the implementation with Spring Boot + web, using regular #PostMapping which has "consumes" and "produces" set to application/xml and application/json, respectively. The usual flow works perfectly fine. Now, I stumbled upon the issue of validating the incoming payload. What I figured out:
1) I have to validate the payload before it is converted (marshalled) to an object.
2) Once validated, I have to either:
allow further processing
stop any further processing, write the error object to the response and set the status code to 400 Bad request
My approaches were:
1) using a RequestBodyAdvice, more specifically the beforeBodyRead method implementation. I had the following issue here: I don't know how to write anything to the output in case the validation fails.
2) using a Filter (I've extended OncePerRequestFilter) - fortunately, I can read the request (request.getInputStream()) and write to the response (response.getOutputStream()).
However, how can I do the selective filtering (as mentioned, I only want to validate one single endpoint)?
Are there any other alternatives for placing the incoming request XSD validation? Is spring-web the appropriate choice here? Would you recommend some other library / framework?
To validate xml against xsd schema, my preference is XML Beans. It is very easy to use. Other options are JABX, Castor. Take a look at Java to XML conversions?.
You will need to jar using xsd schmema and will need to put it in the classpath of your application so that it's classes are available for you for validation. Please take a look at this blog.
You can use validation API as mentioned here.
I would prefer to write validation code in the aspect so that it can be reused with other APIs.
If validation fails, throw valid exception from the aspect itself.
If validation is passed, process your input string that you receive.
Please let us know if you need any more information.
I'm attempting to simulate a login call with JMeter 2.11 to a service that uses a binary format. I've created an Http Request with the appropriate settings, except for the body data. I need to POST raw binary data.
According to the docs here, I should be able to set the file path for exactly one file, with no parameter name, and no other content in the Body Data, and have it place the data in the request body.
If it is a POST or PUT or PATCH request and there is a single file whose 'Parameter name' attribute (below) is omitted, then the file is sent as the entire body of the request, i.e. no wrappers are added. This allows arbitrary bodies to be sent. This functionality is present for POST requests after version 2.2, and also for PUT requests after version 2.3.
However, when I run the test, the POST Data is empty.
I have tried the extra set of plugins for JMeter as well, but alas, I'm stuck. The loaded file has 145 bytes of data, and the request shows that the content-length is 0. What am I missing?
The Http Request
The result
Update 1
To clarify, I am NOT attempting to send a file, I'm attempting to send a binary encoded message as raw POST data.
Switch back to Parameters tab not Post body.
See:
http://jmeter.apache.org/usermanual/component_reference.html#HTTP_Request
Yoy could try recoring to see how the resuest look like.
This is my solution,maybe not best, but it works fine:
1st step :
You should write your binary data to a file (assume it's name is
FILENAME);
2nd step :
For your http request sampler,Yout should put ${FILENAME} under file
path in the "Send Files with the request" section (while leaving its
paramter name empty and specifying an encoding (for binary, it is
application/binary)).
Hope it helps.
Refer to this article