I am developing a java based web application in spring-boot where I am sending http-header from server to client end.
Server-side I have used Spring-boot in form of REST API, whereas at client end we have simple plain HTML5/Angular framework.
My query is, while I am sending any header from server then at client end I always get it into lowercase vs actual.
For example, I am setting header something like,
header.add("KK-ACTUAL-VALUE", "sfsfjDFFDHTsdfJKKA");
At client end, it gives,
kk-actual-value : "sfsfjDFFDHTsdfJKKA"; (Header name converts into lower case!)
Now, the question is, how to prevent it ?
I want to have the same header name what is actually passed by Server/API.
Looks like tomcat container turns the headers to lowercase. Since spring boot uses an embedded tomcat is been affected by it. More info here: https://github.com/mitre/HTTP-Proxy-Servlet/issues/65
As the Http standard specifies:
Each header field consists of a case-insensitive field name followed
by a colon (":") ...
https://www.rfc-editor.org/rfc/rfc7230#section-3.2
To make the header case-sensitive you can set it to the MultiValueMap, and then pass it as a parameter to the HttpHeaders constructor.
The code snippet can be something like this:
MultiValueMap <String, String> headers = new LinkedMultiValueMap<>();
headers.add("KK-ACTUAL-VALUE", "sfsfjDFFDHTsdfJKKA");
HttpHeaders responseHeaders = new HttpHeaders(headers);
Related
We are using Spring Boot 2.1.4 and the micrometer-registry-prometheus dependency to capture metrics.
We have a case where a Spring Boot service uses a RestTemplate to call out to another service. The metrics being generated by this call contain the actual values in the URI instead of the templated values.
For example, in the /actuator/prometheus endpoint, I see entries like this:
http_client_requests_seconds_count{clientName="someClient",method="GET",status="200",uri="/person/lookup?firstName=Tony&lastName=Soprano",} 1.0
Based on the documentation, I would expect to see the variable names rather than the values, like this:
http_client_requests_seconds_count{clientName="someClient",method="GET",status="200",uri="/person/lookup?firstName={firstName}&lastName={lastName}",} 1.0
Is there a way to get the default http.client.requests metric values to use the templated values for the URI tag?
The Spring documentation at https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-features.html#production-ready-metrics-http-clients says this about the uri tag:
Request’s URI template prior to variable substitution, if possible (for example, /api/person/{id})
How do we make it possible for the variable substitution to take place?
I assume you are using the RestTemplateBuilder to build your RestTemplate as otherwise you wouldn't be getting the metrics registered.
Are you actually passing a templated url into RestTemplates exchange methods and pass along the params for subsbitution? Works-for-me on 2.1.4.RELEASE and 2.2.1.RELEASE.
template.getForObject("http://localhost:" + this.serverPort + "/hello/{id}",
String.class, Collections.singletonMap("id", "loop"));
Results in:
http_client_requests_seconds_count{application="micrometered2",clientName="localhost",method="GET",outcome="SUCCESS",status="200",uri="/hello/{id}",} 23.0
I have a Spring application that receives a request like http://localhost/foo?email=foo+bar#example.com. This triggers a controller that roughly looks like this:
#RestController
#RequestMapping("/foo")
public class FooController extends Controller {
#GetMapping
public void foo(#RequestParam("email") String email) {
System.out.println(email)
}
}
By the time I can access email, it's been converted to foo bar#example.com instead of the original foo+bar#example.com. According to When to encode space to plus (+) or %20? this should only happen in requests where the content is application/x-www-form-urlencoded. My request has a content type of application/json. The full MIME headers of the request look like this:
=== MimeHeaders ===
accept = application/json
content-type = application/json
user-agent = Dashman Configurator/0.0.0-dev
content-length = 0
host = localhost:8080
connection = keep-alive
Why is Spring then decoding the plus as a space? And if this is the way it should work, why isn't it encoding pluses as %2B when making requests?
I found this bug report about it: https://jira.spring.io/browse/SPR-6291 which may imply that this is fixed on version 3.0.5 and I'm using Spring > 5.0.0. It is possible that I may misinterpreting something about the bug report.
I also found this discussion about RestTemplate treatment of these values: https://jira.spring.io/browse/SPR-5516 (my client is using RestTemplate).
So, my questions are, why is Spring doing this? How can I disable it? Should I disable it or should I encode pluses on the client, even if the requests are json?
Just to clarify, I'm not using neither HTML nor JavaScript anywhere here. There's a Spring Rest Controller and the client is Spring's RestTemplate with UriTemplate or UriComponentsBuilder, neither of which encode the plus sign the way Spring decodes it.
Original Answer
You are mixing 2 things, a + in the body of the request would mean a space when header has application/x-www-form-urlencoded. The body or content of the request would be dependent on the headers but a request can just have a url and no headers and no body.
So the encoding of a URI cannot be controlled by any headers as such
See the URL Encoding section in https://en.wikipedia.org/wiki/Query_string
Some characters cannot be part of a URL (for example, the space) and some other characters have a special meaning in a URL: for example, the character # can be used to further specify a subsection (or fragment) of a document. In HTML forms, the character = is used to separate a name from a value. The URI generic syntax uses URL encoding to deal with this problem, while HTML forms make some additional substitutions rather than applying percent encoding for all such characters. SPACE is encoded as '+' or "%20".[10]
HTML 5 specifies the following transformation for submitting HTML forms with the "get" method to a web server.1 The following is a brief summary of the algorithm:
Characters that cannot be converted to the correct charset are replaced with HTML numeric character references[11]
SPACE is encoded as '+' or '%20'
Letters (A–Z and a–z), numbers (0–9) and the characters '*','-','.' and '_' are left as-is
All other characters are encoded as %HH hex representation with any non-ASCII characters first encoded as UTF-8 (or other specified encoding)
The octet corresponding to the tilde ("~") is permitted in query strings by RFC3986 but required to be percent-encoded in HTML forms to "%7E".
The encoding of SPACE as '+' and the selection of "as-is" characters distinguishes this encoding from RFC 3986.
And you can see the same behaviour on google.com as well from below screenshots
Also you can see the same behaviour in other frameworks as well. Below is an example of Python Flask
So what you are seeing is correct, you are just comparing it with a document which refers to the body content of a request and not the URL
Edit-1: 22nd May
After debugging it seems the decoding doesn't even happen in Spring. I happens in package org.apache.tomcat.util.buf; and the UDecoder class
/**
* URLDecode, will modify the source.
* #param mb The URL encoded bytes
* #param query <code>true</code> if this is a query string
* #throws IOException Invalid %xx URL encoding
*/
public void convert( ByteChunk mb, boolean query )
throws IOException
{
int start=mb.getOffset();
And below is where the conversion stuff actually happens
if( buff[ j ] == '+' && query) {
buff[idx]= (byte)' ' ;
} else if( buff[ j ] != '%' ) {
This means that it is an embedded tomcat server which does this translation and spring doesn't even participate in this. There is no config to change this behaviour as seen in the class code. So you have to live with it
SPR-6291 fixed this problem in v3.0.5 but this remains unresolved in some other cases like SPR-11047 is still unresolved. While SPR-6291's priority was Major, SPR-11047's priority is Minor.
I faced this problem when I was working on REST API in old Spring last year. There are multiple ways we can get data in Spring controller. So two of them are via #RequestParam or #PathVariable annotation
As others mentioned I think its spring's internal issue and does not specifically belong to URL encoding because I was sending data over POST request but it is somewhat encoding problem. But I also agree with others as now it remains problematic only in URL.
So there are two solutions I know:
You can use #PathVariable instead of #RequestParam because as of SPR-6291 this plus sign issue is fixed in #PathVariable and still remains open for #RequestParam as SPR-11047
My version of spring was not even accepting plus sign via #PathVariable annotation, so this is how I overcome the problem (I don't remember it step by step but it will give you hint).
In your case you can get the fields via JS and escape the plus sign before sending a request. Something like this:
var email = document.getElementById("emailField").value;
email = email.replace('+', '%2B');
If you have this request:
http://localhost/foo?email=foo+bar#example.com
then the original is foo bar#example.com. If you say the original should be foo+bar#example.com then the request should be:
http://localhost/foo?email=foo%2Bbar#example.com
So Spring is working as supposed to. Maybe on client you should check if the URI is properly encoded. The client-side URL encoding is responsible for building a correct HTTP request.
See encodeURI() if you generate the request in JavaScript or uriToString() if you generate the request in Spring.
Build your request string (the part after ?), without any encoding, with unencoded values like foo+bar#email.com, and only in the end, before actually using it in GET, encode all of it with whatever is available on the client platform. If you want to use POST then you should encode it according to the MIME type of your choice.
I made a Spring REST application where you can perform CRUD operations based on HTTP methods of POST, PUT, GET, DELETE. I have the typical URI template of
http://host/root/{id}/{name}/{address} and etc.
We have a client who is accessing this REST service. Apparently they are sending parameters for multi-word name and address in the following form:
http://host/root/11/John+Smith/10+Las+Vegas+USA
They are using the HTML encoding scheme based on application/x-www-form-urlencoded type. According to the article in Wikipedia
The application/x-www-form-urlencoded
type
The encoding used by default is based
on a very early version of the general
URI percent-encoding rules, with a
number of modifications such as
newline normalization and replacing
spaces with "+" instead of "%20". -
http://en.wikipedia.org/wiki/Percent-encoding
However it appears the standard URL encoding scheme is to use %20 in replacing spaces in URI templates. Which one is correct?
My Spring REST automatically converts %20 to spaces. It's interpreted correctly. I'm using Spring 3.0.4. When + is met by my REST service, it's accepted as is. Of course when I put validation to exclude +, it is indeed excluded as expected.
Am I within standards or are there such double standards? Or is the client using an ancient scheme?
The point is that application/x-www-form-urlencoded can be used only in request parameters, whereas percent encoding is also supported in a path.
So,
http://host/root/11/?name=John+Smith&address=10+Las+Vegas+USA
is fine and will be properly decoded by Spring MVC, but
http://host/root/11/John+Smith/10+Las+Vegas+USA
is wrong and Spring MVC doesn't decode it, because the following form should be used instead:
http://host/root/11/John%20Smith/10%20Las%20Vegas%20USA
I am using the Spring RestTemplate to make calls to a Apache Solr index. I form a request string manually and don't supply any intentional {variable_name} template expansion variables. Part of the query is the term {!lucene q.op=OR}. Unfortunately this gets processed by the URITemplate engine as part of a restTemplate.getForObject call.
Ideally i would like to stop this processing. Is there away of escaping the { } characters so that URITemplate doesn't process them? I have tried encoding the characters but RestTemplate assumes a non-encoded string so they are encoded twice and cause a 400: Bad Request on the backend.
Sample URL:
http://localhost/solr/select?q={!lucene
q.op=OR}se_genbanklocus:*
se_gb_create:* se_gb_update:*
se_clone_name:*
se_sample_tissue:*&facet=true&facet.limit=3&facet.mincount=1&facet.field=se_sample_tissue&facet.field=se_sample_tissue_name&facet.field=se_sample_tissue_code&facet.field=se_sample_tissue_class&facet.field=se_nuc_acid_type&facet.field=ssam_sample_georegion&start=0&rows=10
I've found a work around in which i can use the template to expand one variable which contains the offending {!lucene q.op=OR}
restTemplate.getForObject(solrServer+"select?{query}" , String.class, requestString );
The problem here is that you're using RestTemplate for something it's not designed for. The sample URL you gave is not a REST-style URL, it's just a mass of query parameters, using encoded characters that you're not going to find in a REST scheme, hence the difficulty with unwanted substitutions.
How about using the overloaded method that accepts a URI?
Is it possible to use the Spring validation framework with Spring MVC to validate the presence and value of an HTTP request header?
To check the presence of a request header, you don't need the validation framework. Request header parameters are mandatory by default, and if a mandatory header is missing in a request, Spring MVC automatically responds with 400 Bad Request.
So the following code automatically checks the presence of the header "Header-Name"...
#PostMapping("/action")
public ResponseEntity<String> doAction(#RequestHeader("Header-Name") String headerValue) {
// ...
}
... and if the header shall be optional, the annotation would need to be replaced by:
#RequestHeader(name = "Header-Name", required = false)
To check the value of a request header, the Spring validation framework can be used. To do this, you need to
Add #Validated to the controller class. This is a workaround needed until this feature is implemented.
Add the JSR-303 annotation to the request header parameter, e.g.
#RequestHeader("Header-Name") #Pattern(regexp = "[A-Za-z]*") String headerValue
Note however that this will result in a 500 in case of an invalid header value. Check this question for how to also get the correct status code (i.e. 400) for this case.
I don't see how this would be possible, since the validation framework only operates on your domain objects, not on the HTTP request itself. Specifically, the Validator interface doesn't specify any methods that take the HttpServletRequest object, which is what you'd need to have access to in order to grab the headers and test them.
Using the validation framework feels like the wrong solution to whatever problem you're trying to solve, especially since it's hard to know how there'd be a unique HTTP request header for a given form submission. Are you looking to test for an HTTP header that should always be present in requests to your app? Then you might want to consider implementing a HandlerInterceptor, which will intercept and process all requests to pages that you've mapped in any HanderMappings. Are you looking to test for an HTTP header that should always be present in any page view of your app? Then you'd want to implement a Filter, which operates outside of the context of Spring MVC.