For example, if the post url is:
http://www.wolf.com/pcap/search?stime={stime}&etime=${etime}&bpf=${bpf}
then can we do this:
Map<String, String> vars = new HashMap<String, String>();
vars.put("bpf", bpf);
...
responseString = restTemplate.postForObject(url, null, String.class,vars);
If bpf is a String, then is there a limitation of the size of bpf? Can it be any size?
Unfortunately the answer is: "It depends".
More precisely: as you append the bpf as a parameter to the URL it does not really matter if you are doing a POST or a GET. Sometimes there are restrictions on the length of an URL a server will handle, but that depends on what the server accepts, and cannot be determined from the RestTemplate, which is the client.
For example if the server you send the REST request to is a tomcat, then the maximal value of the complete header (URL, HTTP-Header etc) is by default 8kB for tomcat 6.0 or higher; see e.g. https://serverfault.com/questions/56691/whats-the-maximum-url-length-in-tomcat
Just in case if you have control over the server side, too, you can change the expected interface by not sending the bpf as parameter, but as request body, like:
Map<String, String> vars = new HashMap<String, String>();
// vars.put("bpf", bpf); <--- not needed
responseString = restTemplate.postForObject(url, bpf, String.class, vars);
(and then of course get the bpf on the server from the request body instead).
Otherwise you are out of luck and have to limit the length of the URL. Maybe use a proxy or network sniffer to see what extra Headers are actually send and subtract that from the 8kB limit to get the maximal length of the URL.
Related
We have a Controller running on tomcat 8.5.32 which receives a POST request with query params
/{path_param}/issue?title=4&description=5
request body is empty
Then controller redirects this request to Spring Boot microservice with tomcat 9.0.27.
At line
CloseableHttpResponse result = httpClient.execute(request);
request.getURI().getQuery() equals&title=1&description=2
But when it arrives to microservice parameters are duplicated (title=[4,4]&description=[5,5]).
This is the code which redirects request to microservice
private static <T, U> T executePostRequest(String url, U body, HttpServletRequest httpServletRequest, Function<String, T> readValueFunction) {
try (CloseableHttpClient httpClient = HttpClientBuilder.create().build()) {
URIBuilder uriBuilder = new URIBuilder(url);
httpServletRequest.getParameterMap().forEach((k, v) -> Arrays.stream(v).forEach(e -> uriBuilder.addParameter(k, e)));
HttpPost request = new HttpPost(uriBuilder.build());
CloseableHttpResponse result = httpClient.execute(request);
String json = EntityUtils.toString(result.getEntity(), "UTF-8");
handleResultStatus(result, json);
return readValueFunction.apply(json);
} catch (IOException | URISyntaxException e) {
...
}
}
I found that there was similar issue with jetty and it was fixed but did not find anything related to tomcat - and how it can be fixed.
I saw also this topic whith suggestion how to handle duplicated parameters in spring boot but i am wondering if anyone else experienced same issue and how did you resolve it if yes.
It's not a bug, it's a feature present in every servlet container.
The Servlet API does not require for the request parameters to have unique names. If you send a POST request for http://example.com/app/issue?title=1&description=2 with a body of:
title=3&description=4
then each parameter will have multiple values: title will have values 1 and 3, while description will have values 2 and 4 in that order:
Data from the query string and the post body are aggregated into the request
parameter set. Query string data is presented before post body data. For example, if
a request is made with a query string of a=hello and a post body of a=goodbye&a=
world, the resulting parameter set would be ordered a=(hello, goodbye, world).
(Servlet specification, section 3.1)
If you want to copy just the first value of the parameters use:
httpServletRequest.getParameterMap()//
.forEach((k, v) -> uriBuilder.addParameter(k, v[0]));
The url-string contains a back-slash character that needs to be encoded. The url string is as follows.
String folder = "\\Foo\\Bar\\"; // some folder search path.
String urlString= "http://localhost:8081/certificates/?mypath=%5CFoo%5CBar%5C" // (after encoding)
Here I use Spring RestTemplate to do a GET request. I setup a mock-server to examine the request in detail (mock server setup using Mulesoft, if u must know!).
ResponseEntity<String> responseEntity = api.exchange(urlString, HttpMethod.GET, new HttpEntity<>(new HttpHeaders()), String.class);
Here I use plain vanilla Java URLConnection to perform the request. Attached image with detailed request snapshot.
// 2. Plain vanilla java URLConnection. "result.toString()" has certificate match.
StringBuilder result = new StringBuilder();
URL url = new URL(urlString);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestProperty("X-Venafi-Api-Key", apiKey);
conn.setRequestMethod("GET");
BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line;
while ((line = rd.readLine()) != null) {
result.append(line);
}
rd.close();
System.out.println(result.toString());
In the images, you can see that the queryString value is different for these two requests. One of them shows \\ while the other shows %5C, although the parsed parameter value for myPath is still the same.
I am having to deal with an api that seems to work if-and-only-if the queryString looks like the former (i.e. "\\"). Why does the parsed queryString for Spring show "%5C" while this value shows double-backslash for requests originating from plain Java, curl, and even a simple browser?
What baffles me EVEN more, is that just about everything about the two HTTP Requests are IDENTICAL! And yet, why does the queryString/requestUri parse differently for these two requests? Shouldn't it be that a HTTP GET method is completely defined by its header contents and the requestUri? What am I missing to capture in these two GET requests?
Lots of questions. Spent an entire day, but at least I could verify that the way the requestUri/queryString is parsed seems to align with how the remote api-server responds.
Thanks.
Did some digging around the following morning. Turn out, with
ResponseEntity<String> responseEntity = api.exchange(urlString, HttpMethod.GET, new HttpEntity<>(new HttpHeaders()), String.class);
You should NOT have the "urlString" already encoded. The 'exchange' method does that encoding for you under-the-hood.
I am currently testing one of my services with Spring boot test.The service exports all user data and produces a CSV or PDF after successful completion. A file is downloade in browser.
Below is the code i have wrote in my test class
MvcResult result = MockMvc.perform(post("/api/user-accounts/export").param("query","id=='123'")
.contentType(MediaType.APPLICATION_JSON_VALUE)
.accept(MediaType.APPLICATION_PDF_VALUE)
.content(TestUtil.convertObjectToJsonBytes(userObjectDTO)))
.andExpect(status().isOk())
.andExpect(content().contentType(MediaType.APPLICATION_PDF_VALUE))
.andReturn();
String content = result.getResponse().getContentAsString(); // verify the response string.
Below is my resource class code (call comes to this place)-
#PostMapping("/user-accounts/export")
#Timed
public ResponseEntity<byte[]> exportAllUsers(#RequestParam Optional<String> query, #ApiParam Pageable pageable,
#RequestBody UserObjectDTO userObjectDTO) {
HttpHeaders headers = new HttpHeaders();
.
.
.
return new ResponseEntity<>(outputContents, headers, HttpStatus.OK);
}
While I debug my service, and place debug just before the exit, I get content Type as 'application/pdf' and status as 200.I have tried to replicate the same content type in my test case. Somehow it always throws below error during execution -
java.lang.AssertionError: Status
Expected :200
Actual :406
I would like to know, how should i inspect my response (ResponseEntity). Also what should be the desired content-type for response.
You have problem some where else. It appears that an exception/error occurred as noted by application/problem+json content type. This is probably set in the exception handler. As your client is only expecting application/pdf 406 is returned.
You can add a test case to read the error details to know what exactly the error is.
Something like
MvcResult result = MockMvc.perform(post("/api/user-accounts/export").param("query","id=='123'")
.contentType(MediaType.APPLICATION_JSON_VALUE)
.accept(MediaType.APPLICATION_PROBLEM_JSON_VALUE)
.content(TestUtil.convertObjectToJsonBytes(userObjectDTO)))
.andExpect(status().isOk())
.andExpect(content().contentType(MediaType.APPLICATION_PROBLEM_JSON_VALUE))
.andReturn();
String content = result.getResponse().getContentAsString(); // This should show you what the error is and you can adjust your code accordingly.
Going forward if you are expecting the error you can change the accept type to include both pdf and problem json type.
Note - This behaviors is dependent on the spring web mvc version you have.
The latest spring mvc version takes into account the content type header set in the response entity and ignores what is provided in the accept header and parses the response to format possible. So the same test you have will not return 406 code instead would return the content with application json problem content type.
I found the answer with help of #veeram and came to understand that my configuration for MappingJackson2HttpMessageConverter were lacking as per my requirement. I override its default supported Mediatype and it resolved the issue.
Default Supported -
implication/json
application*/json
Code change done to fix this case -
#Autowired
private MappingJackson2HttpMessageConverter jacksonMessageConverter;
List<MediaType> mediaTypes = new ArrayList<>();
mediaTypes.add(MediaType.ALL);
jacksonMessageConverter.setSupportedMediaTypes(mediaTypes);
406 means your client is asking for a contentType (probably pdf) that the server doesn't think it can provide.
I'm guessing the reason your code is working when you debug is that your rest client is not adding the ACCEPT header that asks for a pdf like the test code is.
To fix the issue, add to your #PostMapping annotation produces = MediaType.APPLICATION_PDF_VALUE see https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/web/bind/annotation/PostMapping.html#produces--
I've upgrade Spring version from 4.0.x to 4.2.3 (4.2.4 just now), suddenly, "required = false" property from #RequestBody annotation does not work as expected (as it did before the version change).
Server response with HttpStatus 415 - Unsupported Media Type.
The controller method (same for every Spring versions).
#RequestMapping(value = "/entity/{entityId}/{from}/{size}", method = RequestMethod.POST)
#ResponseBody
#JsonView(ToShowIn.App.class)
public ResponseEntity<?> getActiveEntityPaged(
#PathVariable int entityId, #PathVariable int from, #PathVariable int size,
#RequestBody(required = false) EntityFilterParam entityFilterParam) {
...
}
Really, it's not problem because from the client, avoiding send null, for example, setting empty EntityFilterParam instance is enough to fix it. But I would like to understand if this bothering issue is because a wrong concept from me or rather is due for a bug from the new Spring versions.
UPDATE#1
Versions
<spring-version>4.2.4.RELEASE</spring-version>
<jackson-2-version>2.3.2</jackson-2-version>
Request is from Android device, working with spring-android library (AKA RestTemplate).
Generic method, works on 4.0.x version, where EntityFilterParam can be null.
super.doRestList(
new HttpEntity<EntityFilterParam>(EntityFilterParam, this.getHttpHeaders()),
HttpMethod.POST,
urlBuilder);
On 4.2.3 and 4.2.4, to fix the issue, EntityFilterParam can't be null. -> HttpStatus 415 - Unsupported Media Type. To fix it.
super.doRestList(
new HttpEntity<EntityFilterParam>((EntityFilterParam != null) ? EntityFilterParam : new EntityFilterParam(), this.getHttpHeaders()),
HttpMethod.POST,
urlBuilder);
Inside doRestList, RestTemplate request is performed.
final ResponseEntity<E[]> responseEntity =
(ResponseEntity) super.getRestTemplate()
.exchange(urlBuilder.getURL(), httpMhetod,
requestEntity, this.entityArray.getClass());
Where, urlBuilder.getURL() (URL string, here #PathVariables are working fine) and this.getHttpHeaders() return the next HttpHeaders configuration.
RestFactory.httpHeaders = new HttpHeaders();
RestFactory.httpHeaders.setAuthorization(this.getAuthorization(user));
RestFactory.httpHeaders.set(HttpHeaders.CONNECTION, "Close"); // Switch off keep-alive.
RestFactory.httpHeaders.set(HttpHeaders.ACCEPT_LANGUAGE, "en-US,en;q=0.8");
RestFactory.httpHeaders.set(HttpHeaders.ACCEPT, MediaType.APPLICATION_JSON_VALUE);
super.getRestTemplate() is the singleton method which returns the RestTemplate instance set with my own CloseableHttpClient configuration.
I know HttpHeaders.CONTENT_TYPE for application/json is missing, but is consciously, in fact, it works when the http body payload is not empty, and fails when it is.
When the error occurs, 415 is performed before the flow of the request goes inside the controller method, therefore, any operation is doesn't performed.
UPDATE#2
Another interesting point, Jackson2 serializes the null value as "null" string, then the body payload is not being null, is filled with "null".
I've tried reproducing this exact issue, without much success.
Do you confirm that with a curl command the problem still exists?
Something like curl -vvv -X POST https://example.com/entity/12/50/10 (if so, paste here the output)
I'm not 100% about this for RestTemplate in spring-android, but if no content type is provided, the template adds by default a "Content-Type: application/x-www-form-urlencoded". If this is the case, configuring a "DEBUG" log level for "org.springframework.web" should show something like
[INFO] [talledLocalContainer] org.springframework.web.HttpMediaTypeNotSupportedException:
Content type 'application/x-www-form-urlencoded' not supported
I think you should set the Content-Type anyway. You mentioned this fails when you do - could you elaborate a bit on that?
I want to do an integration test for the below action.
How can I pass my requestDto object in the integration test?
Neither the GetAsync nor SendAsync method has an overload parameter to pass a custom object to the server.
[Route("{startDate:datetime}")]
[HttpGet]
public HttpResponseMessage Get(DateTime startDate, [FromBody]LessonplannerGetRequest request)
{
request.StartDate = startDate;
var lessonplannerResponse = _service.GetPeriodsByWeekStartDate(request);
return Request.CreateResponse<LessonplannerResponse>(HttpStatusCode.OK, lessonplannerResponse);
}
[Test]
public void Get_Lessons_By_Date()
{
// Arrange
var request = new HttpRequestMessage(HttpMethod.Get, _server.BaseAddress + "/api/lessonplanner/2014-01-14");
var myRequestDto = new LessonplannerGetRequest();
// Act => QUESTION: HOW do I pass the myRequestDto ???
var response = _client.SendAsync(request, new CancellationToken()).Result;
// Assert
Assert.That(response.StatusCode == HttpStatusCode.OK);
}
UPDATE
As Darrel Miller said:"Technically HTTP says you can send a body, it just says the body doesn't mean anything and cannot be used. HttpClient won't let you send one."
I post here my integration test with HttpClient doing a Get request with complex type + FromBody:
// Arrange
var request = new HttpRequestMessage(HttpMethod.Get, _server.BaseAddress + "/api/lessonplanner/2014-01-14");
var myRequestDto = new LessonplannerGetRequest{ FirstDayOfWeek = DayOfWeek.Sunday, SchoolyearId = 1, StartDate = DateTime.Today};
request.Content = new ObjectContent<LessonplannerGetRequest>(myRequestDto, new JsonMediaTypeFormatter());
request.Content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
_client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
// Act
var response = _client.SendAsync(request, new CancellationToken()).Result;
// Assert
Assert.That(response.StatusCode == HttpStatusCode.OK);
Of course is this is not the Http way some might consider doing it differentlly sending complex type via FromUri/query string.
HTML specifications says you cannot send a GET with a body.
HTTP specs allows it.
WebAPI allows it, because it is a service/REST and implements HTTP but not HTML, but many clients and browser won't allow it because they implement both specs and try to be strict.
As for the specifications (RFC1866, page 46; HTML 4.x section 17.13.3) itself, it states:
If the method is "get" and the action is an HTTP URI, the user agent takes the value of action, appends a `?' to it, then appends the form data set, encoded using the "application/x-www-form-urlencoded" content type.
(e.g. if you do a <form> with GET, it will parse all the form params and set them in the query string ?a=b).
In term of pure HTTP and in the context of REST services, nothing prevents that behavior, but not all clients will be able to handle it. It's mostly a best-practice advise when it comes to REST/WebAPI to not handle body data from HttpGet, only URI data (the opposite, POST /action?filter=all is usually tolerated for metadata/action qualifiers, but that's another discussion).
So yeah, it's at your own risk, even if used only internally. As not all clients handle it (e.g. HttpRequestMessage), so you might run into trouble like you have.
You should NOT pass a GET body with HTTPClient.