Am working with Reactive Webclient in Spring, and am having some difficulty determining why am getting the below response format from a GET request:
�\jj��T{�8��?�t(O�"V$�#��)���5�����~?��J�t"�� ��EEΕ7�b��,�u�i�=��k���3�ӡ�������m,��ꔁu�T낙*��{�DZ�l�`����{� ��M��+mB)��E{�UtG���ʘJ#ؠ�]j<u�?6�`h���.���𢚾�n��?�ʋ*��#%�w'<L��d�Ly����R�>���$�O�����—o�}�� �����v���{���?����r�;
This is the code am using below for the request:
return webClient
.get()
.uri(URI.create(uri))
.accept(MediaType.TEXT_HTML, MediaType.APPLICATION_XHTML_XML)
.acceptCharset(StandardCharsets.UTF_8, StandardCharsets.ISO_8859_1)
.header("Accept-Encoding", "gzip, deflate, br")
.retrieve()
.bodyToMono(String.class);
WebClient setup is as below:
HttpClient client = HttpClient.create()
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
.followRedirect(true)
.secure()
.responseTimeout(Duration.ofMillis(readTimeoutMillis));
return WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(client))
.build();
What am I missing?
Looks like response is encoded, probably gzipped, especially that you have declared to the server, that you are able to decode gzip, deflate and br encodings with
Accept-Encoding: gzip,deflate,br
You can remove .header("Accept-Encoding", "gzip, deflate, br") and see if the server is willing to send you a plain text response.
Related
I am calling an external server using HttpRequestExecutingMessageHandler. I am using a JSON to object transformer to convert the JSON data. But I am getting the following exception.
Caused by: com.fasterxml.jackson.core.JsonParseException: Illegal character ((CTRL-CHAR, code 31)): only regular white space (\r, \n, \t) is allowed between tokens
at [Source: (String)"�
when I checked the headers I found the following.
Transfer-Encoding=chunked
Is this the reason for the exception log?
The outbound gateway and JsonToObjectTransformer are given below:
#ServiceActivator(inputChannel = "channelOutboundRequest")
#Bean
public HttpRequestExecutingMessageHandler outboundGateway() {
final HttpRequestExecutingMessageHandler handler = new HttpRequestExecutingMessageHandler(
endpoint);
handler.setExpectedResponseType(String.class);
handler.setHttpMethod(HttpMethod.POST);
handler.setOutputChannelName("channelResponse");
handler.setAdviceChain(Collections.singletonList(advice()));
return handler;
}
#Bean
#Transformer(inputChannel = "channelResponse", outputChannel = "channelReply")
public JsonToObjectTransformer transformer(ObjectMapper objectMapper) {
final JsonObjectMapper<?, ?> mapper = new Jackson2JsonObjectMapper(objectMapper);
return new JsonToObjectTransformer(DetailsDTO.class, mapper);
}
If the header is causing the issue, how can I handle the response?
Note: If I hit the external server directly using postman, I am getting the response in JSON structure.
I have no idea what's wrong here.. If I use the simple restemplate call like below, it works properly.
JSONObject jsonObject = new JSONObject("{\"code\":\"F001\",\"transactionId\":\"1008566223232\"}");
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON);
headers.setBearerAuth("token");
HttpEntity<String> request =
new HttpEntity<String>(jsonObject.toString(), headers);
String respns = restTemplt.postForObject("http://endpoint", request, String.class);
System.out.println(respns);
JSONObject response = new JSONObject(respns);
One difference I could find was in the response headers. those are given below:
Outbound gateway response headers :-
{Transfer-Encoding=chunked, http_requestMethod=GET, errorChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#d8b195e, Server=nginx, Accept=/, Connection=keep-alive, User-Agent=PostmanRuntime/7.28.0, Host=localhost:8901, Accept-Encoding=gzip, deflate, br, http_statusCode=200 OK, Date=1622533072000, Authorization=Bearer token, replyChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#d8b195e, Cache-Control=no-cache, ETag=W/"1009-5SzdL+uWyY6ZcMWht5dMtm2Sxlc", Content-Encoding=gzip, http_requestUrl=http://inboundurl, id=be07fc8d-d478-5fa9-33e4-61a2b5f92468, Content-Length=207, contentType=application/json;charset=utf-8, Content-Type=application/json, requestFrom=CUSTOM_HEADER, timestamp=1622533092827}
Normal restTemplate call response header
[Server:"nginx", Date:"Tue, 01 Jun 2021 07:34:54 GMT", Content-Type:"application/json; charset=utf-8", Content-Length:"4105", Connection:"keep-alive", Access-Control-Allow-Origin:"*", Content-Security-Policy:"default-src 'self';base-uri 'self';block-all-mixed-content;font-src 'self' https: data:;frame-ancestors 'self';img-src 'self' data:;object-src 'none';script-src 'self';script-src-attr 'none';style-src 'self' https: 'unsafe-inline';upgrade-insecure-requests", X-DNS-Prefetch-Control:"off", Expect-CT:"max-age=0", X-Frame-Options:"SAMEORIGIN", Strict-Transport-Security:"max-age=15552000; includeSubDomains", X-Download-Options:"noopen", X-Content-Type-Options:"nosniff", X-Permitted-Cross-Domain-Policies:"none", Referrer-Policy:"no-referrer", X-XSS-Protection:"0", ETag:"W/"1009-llD9DqxYkEsjyikWajYk+16cb1k""]
Can anyone please help?
After many trials and errors, I found the reason for this. Accept-Encoding=gzip, deflate, br this header in the outbound gateway request is the root cause. the response I am getting is a long string and it is getting compressed because of this header. i added a header filter before the outbound gateway to remove this header.
#Bean
#Transformer(inputChannel = "channelHeaderFilterReq", outputChannel = "channelHeaderFilterRes")
public HeaderFilter filter() {
return new HeaderFilter("Accept-Encoding");
}
Now everything works fine..!!
Below is the code which gives 400 bad request, tried various options with this but always ends up with 400 bad request, however the same works using OkHttp client
HttpHeaders hh = new HttpHeaders();
RestTemplate rt = new RestTemplate();
String reqBody = new ObjectMapper.writeValueAsString(“test msg”);
headers.set(“Accept”, “text/plain”);
headers.set(“content-type”,”application/json”);
headers.set(“authorization”, “Basic xxxxx”);
headers.set(“ibm-mq-rest-csrf-token”,”blank”);
HttpEntity<String> request = new HttpEntity<>(reqBody.toString());
ResponseEntity<Object> result = rt.postForObject(url, request, Object.class);
Below code works using okhttp
Your content does not appear to be JSON, in both cases it appears to be plain text. Therefore the header
Content-Type: text/plain
should be used rather than
Content-Type: application/json
I am unsure why this works in your OKHTTP example, unless perhaps the media type you use to build your post body somehow over-rides your Content-Type header?
I am calling a web service using Spring WebClient (Spring 5.1.3). The service responds with content-type: application/json and content-encoding: gzip
ClientResponse.bodyToMono then fails with the error "JSON decoding error: Illegal character ((CTRL-CHAR, code 31))" which I assume is because the content has not been decoded before trying to parse the JSON.
Here is code snippet (simplified) of how I create the WebClient
HttpClient httpClient = HttpClient.create().secure(sslContextSpec -> sslContextSpec.sslContext(sslContext));
return WebClient.builder().clientConnector(new ReactorClientHttpConnector(httpClient)).build();
I then use the WebClient to make the call:
webClient.get().uri(uri)
.accept(MediaType.APPLICATION_JSON)
.header(HttpHeaders.ACCEPT_ENCODING, "gzip")
.exchange()
The HTTP request has 2 headers:
Accept: application/json
Accept-Encoding: gzip
The response has the following headers:
set-cookie: xxx
content-type: application/json; charset=utf-8
content-length: 1175
content-encoding: gzip
cache-control: no-store, no-cache
By doing the following I am able to manually decode the GZIP content and get valid JSON from the result
webClient.get().uri(uri)
.accept(MediaType.APPLICATION_JSON)
.header("accept-encoding", "gzip")
.exchange()
.flatMap(encodedResponse -> encodedResponse.body((inputMessage, context) ->
inputMessage.getBody().flatMap(dataBuffer -> {
ClientResponse.Builder decodedResponse = ClientResponse.from(encodedResponse);
try {
GZIPInputStream gz = new GZIPInputStream(dataBuffer.asInputStream());
decodedResponse.body(new String(gz.readAllBytes()));
} catch (IOException e) {
e.printStackTrace();
}
decodedResponse.headers(headers -> {
headers.remove("content-encoding");
});
return Mono.just(decodedResponse.build());
}).flatMap(clientResponse -> clientResponse.bodyToMono(Map.class))
This feature is supported natively by the reactor netty client.
You should create HttpClient like this:
HttpClient httpClient = HttpClient.create()
.secure(sslContextSpec -> sslContextSpec.sslContext(sslContext))
.compress(true);
And then there's no need to add the accept encoding request header since it's done for you.
Note that this bit is done by the connector itself when you don't provide a custom HttpClient instance.
I've created a Spring Boot back end with a React front end.
When I'm sending a HTTP request via browser to my backend, I receive a
response with empty headers.
Browser response with empty headers
When I'm sending a HTTP request via the POSTMAN tool, I got filled headers!
POSTMAN response with fullfilled headers
That doesn't make sense to me.
Code: https://github.com/The-Taskmanager/SelfServiceWebwizard
Back end Mapping
#PostMapping("/signin")
public ResponseEntity<?> authenticateUser(#Valid #RequestBody LoginRequest loginRequest) {
Authentication authentication = authenticationManager.authenticate(
new UsernamePasswordAuthenticationToken(
loginRequest.getUsernameOrEmail(),
loginRequest.getPassword()
)
);
SecurityContextHolder.getContext().setAuthentication(authentication);
String jwt = tokenProvider.generateToken(authentication);
HttpHeaders headers = new HttpHeaders();
headers.add("Mustermann", "Max");
headers.add("Content-Type", "application/json");
headers.add("Authorization", new JwtAuthenticationResponse(jwt).getAccessToken());
return ResponseEntity.ok().headers(headers).body(new JwtAuthenticationResponse(jwt));
}
Front end request
login(username: string, password: string) {
fetch('http://localhost:8080/api/auth/signin', {
method: 'POST',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify({
"usernameOrEmail": username,
"password": password
})
}).then(response => {
console.log(response);
console.log(response.text());
console.log(response.headers);
})
.catch(error => console.error(error));
}
UPDATE1:
Added this method but headers in browser still empty:
CorsConfigurationSource corsConfigurationSource() {
final UrlBasedCorsConfigurationSource source = new UrlBasedCorsConfigurationSource();
CorsConfiguration config = new CorsConfiguration().applyPermitDefaultValues();
source.registerCorsConfiguration("/**", config);
return source;
}
UPDATE2:
When I request with React, Spring Boot shows me this ERROR:
ERROR 8876 --- [nio-8080-exec-4] c.s.jwt.JwtAuthenticationEntryPoint: Responding with unauthorized error. Message - full authentication is required to access this resource
Browser console:
When I request with Postman, Spring Boot shows no ERROR.
UPDATE3:
Request-header send by React:
{
host=[localhost:8080],
content-type=[application/json],
cache-control=[no-cache],
user-agent=[Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0],
accept-encoding=[gzip, deflate],
content-length=[58],
referer=[http://localhost:8080/],
origin=[http://localhost:8080],
dnt=[1],
connection=[keep-alive],
accept-language=[de,en-US;q=0.7,en;q=0.3],
accept=[*/*]
}
Request-header send by POSTMAN from POSTMAN console:
host:"localhost:8080"
content-type:"application/json"
cache-control:"no-cache"
user-agent:"PostmanRuntime/7.1.5"
accept-encoding:"gzip, deflate"
content-length:68
postman-token:"b00fd7a8-bd34-4a32-8681-990b04012e3b"
cookie:"JSESSIONID=2CAEC9280432DD13AABA53B73B7874AC"
accept:"*/*"
Solved by myself.
Changed code in front end's fetch method
console.log(response.headers)
// changed to:
console.log(response.headers.get('Authorization')
So the headers were there all the time but I couldn't manage to log them in browser console with React.
Nevertheless there is the Spring Boot (full authentication) and Browser (401) error in UPDATE2. Don't know where this is coming from.
trying to send a Fiddler Post request to my C# API as follows (this is my dev environment using VS2012). However, my request object is null in C#. In the parsed tab of the composer tab. My post URL: http://localhost:33218/api/drm
User-Agent: Fiddler/4.4.9.2 (.NET 4.0.30319.34209; WinNT 6.1.7601 SP1; en-US; 4xAMD64)
Pragma: no-cache
Accept-Language: en-US
Host: localhost:33218
Accept-Encoding: gzip, deflate
Connection: Close
Content-Length: 80
Request Body:
&sid=f7f026d60bb8b51&riskMeasureName=RMTest
And here's the C# API method:
// POST api/drm
public HttpResponseMessage Post([FromBody]JObject drmObject)
{
string sid = drmObject.GetValue("sid").ToString();
string riskMeasCategory = drmObject.GetValue("riskMeasureName").ToString();
string response = DynAggrClientAPI.insertDRMCategory(sid, riskMeasCategory);
var httpResp = Request.CreateResponse(HttpStatusCode.OK);
httpResp.Content = new StringContent(response, Encoding.UTF8, "application/json");
return httpResp;
}
I can debug in my C# Post() method, but the drmObject is null.
Your advice is appreciated.
You're not sending a content-type, so MVC has no way to tell how to interpret the data.
Your data seems to resemble a form POST, so add the header:
Content-Type: application/x-www-form-urlencoded