Decode content-encoding gzip using Spring WebClient - spring

I am calling a web service using Spring WebClient (Spring 5.1.3). The service responds with content-type: application/json and content-encoding: gzip
ClientResponse.bodyToMono then fails with the error "JSON decoding error: Illegal character ((CTRL-CHAR, code 31))" which I assume is because the content has not been decoded before trying to parse the JSON.
Here is code snippet (simplified) of how I create the WebClient
HttpClient httpClient = HttpClient.create().secure(sslContextSpec -> sslContextSpec.sslContext(sslContext));
return WebClient.builder().clientConnector(new ReactorClientHttpConnector(httpClient)).build();
I then use the WebClient to make the call:
webClient.get().uri(uri)
.accept(MediaType.APPLICATION_JSON)
.header(HttpHeaders.ACCEPT_ENCODING, "gzip")
.exchange()
The HTTP request has 2 headers:
Accept: application/json
Accept-Encoding: gzip
The response has the following headers:
set-cookie: xxx
content-type: application/json; charset=utf-8
content-length: 1175
content-encoding: gzip
cache-control: no-store, no-cache
By doing the following I am able to manually decode the GZIP content and get valid JSON from the result
webClient.get().uri(uri)
.accept(MediaType.APPLICATION_JSON)
.header("accept-encoding", "gzip")
.exchange()
.flatMap(encodedResponse -> encodedResponse.body((inputMessage, context) ->
inputMessage.getBody().flatMap(dataBuffer -> {
ClientResponse.Builder decodedResponse = ClientResponse.from(encodedResponse);
try {
GZIPInputStream gz = new GZIPInputStream(dataBuffer.asInputStream());
decodedResponse.body(new String(gz.readAllBytes()));
} catch (IOException e) {
e.printStackTrace();
}
decodedResponse.headers(headers -> {
headers.remove("content-encoding");
});
return Mono.just(decodedResponse.build());
}).flatMap(clientResponse -> clientResponse.bodyToMono(Map.class))

This feature is supported natively by the reactor netty client.
You should create HttpClient like this:
HttpClient httpClient = HttpClient.create()
.secure(sslContextSpec -> sslContextSpec.sslContext(sslContext))
.compress(true);
And then there's no need to add the accept encoding request header since it's done for you.
Note that this bit is done by the connector itself when you don't provide a custom HttpClient instance.

Related

Invalid HTML response when using Java webclient

Am working with Reactive Webclient in Spring, and am having some difficulty determining why am getting the below response format from a GET request:
�\jj��T{�8��?�t(O�"V$�#��)���5�����~?��J�t"�� ��EEΕ7�b��,�u�i�=��k���3�ӡ�������m,��ꔁu�T낙*��{�DZ�l�`����{� ��M��+mB)��E{�UtG���ʘJ#ؠ�]j<u�?6�`h���.���𢚾�n��?�ʋ*��#%�w'<L��d�Ly����R�>���$�O�����—o�}�� �����v���{���?����r�;
This is the code am using below for the request:
return webClient
.get()
.uri(URI.create(uri))
.accept(MediaType.TEXT_HTML, MediaType.APPLICATION_XHTML_XML)
.acceptCharset(StandardCharsets.UTF_8, StandardCharsets.ISO_8859_1)
.header("Accept-Encoding", "gzip, deflate, br")
.retrieve()
.bodyToMono(String.class);
WebClient setup is as below:
HttpClient client = HttpClient.create()
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
.followRedirect(true)
.secure()
.responseTimeout(Duration.ofMillis(readTimeoutMillis));
return WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(client))
.build();
What am I missing?
Looks like response is encoded, probably gzipped, especially that you have declared to the server, that you are able to decode gzip, deflate and br encodings with
Accept-Encoding: gzip,deflate,br
You can remove .header("Accept-Encoding", "gzip, deflate, br") and see if the server is willing to send you a plain text response.

How to stop Spring MVC from automatically adding the "Content-Length: 0" header to HEAD responses?

I'm specifically trying to test the case where my application doesn't receive a Content-Length header from the server, so I've set up my code not to include that header, but for some reason Spring is including it anyway with a value of 0:
#RequestMapping(value = "/test", method = RequestMethod.HEAD)
public void headTest(HttpServletRequest request, HttpServletResponse response) {
response.addDateHeader("Date", System.currentTimeMillis());
response.addHeader("Accept-Ranges", "bytes");
response.addHeader("Content-Type", "video/mp4");
}
$ curl -I http://myserver.com:8600/test
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Date: Thu, 28 Jul 2022 01:05:11 GMT
Accept-Ranges: bytes
Content-Type: video/mp4
Content-Length: 0
How can I stop Spring from including this header?
Setting a header to null to effectively remove it from the response works for embedded Tomcat and might work for other servers:
response.setHeader("Content-Length", null);

Unable to POST data using TLS 1.2

I am trying to POST data to a server who is just using TLS1.2
When i am running the code i am getting the following response from server.
HttpResponseProxy{HTTP/1.1 415 Unsupported Media Type [Date: Wed, 07 Sep 2016 17:42:57 CST, Server: , Strict-Transport-Security: max-age=63072000; includeSubdomains; preload, Pragma: no-cache, Content-Length: 0, charset: ISO-8859-1, X-ORACLE-DMS-RID: 0, X-ORACLE-DMS-ECID: 343fa6c0-ed24-4003-ad58-342caf000404-00000383, Set-Cookie: JSESSIONID=1ZMIvt1NqrtWpHgHs4mMmYyTPUGTOQgrA9biCE3Dok5v0gDCPXu6!681252631; path=/; secure; HttpOnly;HttpOnly;Secure, Cache-Control: no-store, P3P: policyref="/w3c/p3p.xml", CP="WBP DSP NOR AMT ADM DOT URT POT NOT", Keep-Alive: timeout=5, max=250, Connection: Keep-Alive, Content-Type: text/xml, Content-Language: en] [Content-Type: text/xml,Content-Length: 0,Chunked: false]}
I am using the below code to post the data to server . I am using apache httpcomponents-client-4.5.2.
private static Registry<ConnectionSocketFactory> getRegistry() throws KeyManagementException, NoSuchAlgorithmException {
SSLContext sslContext = SSLContexts.custom().build();
SSLConnectionSocketFactory sslConnectionSocketFactory = new SSLConnectionSocketFactory(sslContext,
new String[]{"TLSv1.2"}, null, SSLConnectionSocketFactory.getDefaultHostnameVerifier());
return RegistryBuilder.<ConnectionSocketFactory>create()
.register("https", sslConnectionSocketFactory)
.register("https", sslConnectionSocketFactory)
.build();
}
public static void main(String[] args) throws Exception
{
PoolingHttpClientConnectionManager clientConnectionManager = new PoolingHttpClientConnectionManager(getRegistry());
clientConnectionManager.setMaxTotal(100);
clientConnectionManager.setDefaultMaxPerRoute(20);
HttpClient client = HttpClients.custom().setConnectionManager(clientConnectionManager).build();
HttpPost request = new HttpPost("https://someserver.com/dataupload");
File file = new File("C://Nible//code//_client//Request.xml");
MultipartEntityBuilder builder = MultipartEntityBuilder.create();
builder.setMode(HttpMultipartMode.BROWSER_COMPATIBLE);
builder.setContentType(ContentType.TEXT_XML);
FileBody fileBody = new FileBody(file);
builder.addPart("my_file", fileBody);
HttpEntity reqEntity = builder.build();
request.setEntity(reqEntity);
HttpResponse response = client.execute(request);
System.out.println(response);
}
Can you please tell me what wrong am i doing?
I tried using below code instead of MultipartEntityBuilder. I am still getting the same error.
EntityBuilder builder = EntityBuilder.create();
builder.setFile(file);
builder.setContentType(ContentType.TEXT_XML);
If i am sending the BLANK REQUEST to server then also i am getting the same error. Blank error means i am not putting any thing in request just
HttpPost request = new HttpPost("https://someserver.com/dataupload");
HttpResponse response = client.execute(request);
I suspect of these lines in your code:
MultipartEntityBuilder builder = MultipartEntityBuilder.create();
builder.setMode(HttpMultipartMode.BROWSER_COMPATIBLE);
builder.setContentType(ContentType.TEXT_XML);
Multipart entities cannot be of content-type xml. They must be one of these types:
multipart/mixed
multipart/alternative
multipart/digest
multipart/parallel
(See RFC 1341 7.2)
I guess you should use one of these content-types for the multipart entity, and set text/xml as the content type of the single part:
FileBody fileBody = new FileBody(file, ContentType.TEXT_XML);
(Another issue is that I don't see necessary to send a multipart for just one file: You could leave out the MultipartEntityBuilder object and build directly a FileEntity.)

How to create a Post request in Fiddler

trying to send a Fiddler Post request to my C# API as follows (this is my dev environment using VS2012). However, my request object is null in C#. In the parsed tab of the composer tab. My post URL: http://localhost:33218/api/drm
User-Agent: Fiddler/4.4.9.2 (.NET 4.0.30319.34209; WinNT 6.1.7601 SP1; en-US; 4xAMD64)
Pragma: no-cache
Accept-Language: en-US
Host: localhost:33218
Accept-Encoding: gzip, deflate
Connection: Close
Content-Length: 80
Request Body:
&sid=f7f026d60bb8b51&riskMeasureName=RMTest
And here's the C# API method:
// POST api/drm
public HttpResponseMessage Post([FromBody]JObject drmObject)
{
string sid = drmObject.GetValue("sid").ToString();
string riskMeasCategory = drmObject.GetValue("riskMeasureName").ToString();
string response = DynAggrClientAPI.insertDRMCategory(sid, riskMeasCategory);
var httpResp = Request.CreateResponse(HttpStatusCode.OK);
httpResp.Content = new StringContent(response, Encoding.UTF8, "application/json");
return httpResp;
}
I can debug in my C# Post() method, but the drmObject is null.
Your advice is appreciated.
You're not sending a content-type, so MVC has no way to tell how to interpret the data.
Your data seems to resemble a form POST, so add the header:
Content-Type: application/x-www-form-urlencoded

How to fix System.ArgumentException in HttpWebResponse?

I am facing this exception when receiving HttpWebResponse for my WindowsPhone app. How am I supposed to fix this. It happens very often but I need to make sure my app doesn't crash if it happens. Please have a look at the screenshot.
My expected response is
Headers:-
HTTP/1.1 500 Internal Server Error
Date: Wed, 28 Nov 2012 06:41:24 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
Keep-Alive: timeout=30
Set-Cookie: ...........; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Internal Server Error:
Json:-
{"status":0,"error_code":1001,"data":{"msg":"Something went wrong. Please try again later. [error code 1001]"}}
It also shows in the InnerException the message as Specified value has invalid HTTP Header characters.
Parameter name: name
Please help. I don't know why webRequest.EndGetResponse(asynchronousResult) is not able to read the response. Is there an alternative?
UPDATE
to start the request:
_webRequest.BeginGetRequestStream(new AsyncCallback(GetReqeustStreamCallback), _webRequest);
private void GetReqeustStreamCallback(IAsyncResult asynchronousResult)
{
if ((!ReqIdEnabled || Network.RequestUniqueId == this.RequestUniqueId))
{
HttpWebRequest webRequest = (HttpWebRequest)asynchronousResult.AsyncState;
// End the stream request operation
using (Stream postStream = webRequest.EndGetRequestStream(asynchronousResult))
{
// Add the post data to the web request
postStream.Write(_postDataInBytes, 0, _postDataInBytes.Length);
//postStream.Dispose();
}
// Start the web request
webRequest.BeginGetResponse(new AsyncCallback(GetResponseCallback), webRequest);
}
}
private void GetResponseCallback(IAsyncResult asynchronousResult)
{
HttpWebRequest webRequest = (HttpWebRequest)asynchronousResult.AsyncState;
try
{
//**throws Exception here when my server returns 503/500 and is not caught by the catch block below**
using (HttpWebResponse response = (HttpWebResponse)webRequest.EndGetResponse(asynchronousResult))
{
ReadResponse(response);
}
}
catch (WebException ee)
{
}
}
Put a breakpoint in your catch block, and look at the lower level stacks, and look for System.Net.ni.dll!System.Net.WebHeaderCollection.this[string].set(string name, string value).
In the local variables, you can see for which particular value, the parse is failing.
For me, It was Google App Engine's custom User-Agent header which was creating the problem.

Resources