I am using Spring Boot to integrate to a third party which can provide gzipped responses and I want to log the size of the compressed response body vs the decompressed so we can audit for billing etc.
I'm using RestTemplate.exchange to make the GET request and the response is coming back correctly however I am unable to retrieve the Content-Length header in the response.
I know that the third party are sending it back because I can see it when I make the same request using Curl:
HTTP/1.1 200 OK
Date: Mon, 04 Feb 2019 15:40:10 GMT
Server: Apache
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 558
Connection: close
Content-Type: application/json
This is my Java code making the request and logging the lengths:
final ResponseEntity<String> responseEntity = restTemplate.exchange(url, GET, entity, String.class);
final String body = responseEntity.getBody();
LOG.info("Payload sizes: compressed={}, uncompressed={}", responseEntity.getHeaders().getContentLength(), body.length());
For the request headers I am setting 'Accept-Encoding: gzip,deflate' as per the documentation for the third party. However when I get the response the call to getContentLength() just returns -1. I've done a lot of searching around on this and tried using a ClientHttpRequestInterceptor but no joy.
Anybody ever had the same issues?
Just in case anyone got same issue as I had, if you are using TestRestTemplate, it will not show Content-Encode when gzip, try use RestTemplate instead
Related
I'm experiencing my client getting logged out after an innocent request to my server. I control both ends and after a lot of debugging, I've found out that the following happens:
The client sends the request with a correct Authorization header.
The server responds with 304 Not Modified without any Authorization header.
The browser serves the full response including an obsolete Authorization header as found in its cache.
From now on, the client uses the obsolete Authorization and gets kicked out.
From what I know, the browser must not cache any request containing Authorization. Nonetheless,
chrome://view-http-cache/http://localhost:10080/api/SearchHost
shows
HTTP/1.1 200 OK
Date: Thu, 23 Nov 2017 23:50:16 GMT
Vary: origin, accept-encoding, authorization, x-role
Cache-Control: must-revalidate
Server: 171123_073418-d8d7cb0 =
x-delay-seconds: 3
Authorization: Wl6pPirDLQqWqYv
Expires: Thu, 01 Jan 1970 00:00:00 GMT
ETag: "zUxy1pv3CQ3IYTFlBg3Z3vYovg3zSw2L"
Content-Encoding: gzip
Content-Type: application/json;charset=utf-8
Content-Length: 255
The funny server header replaces the Jetty server header (which shouldn't be served for security reasons) by some internal information - ignore that. This is what curl says:
< HTTP/1.1 304 Not Modified
< Date: Thu, 23 Nov 2017 23:58:18 GMT
< Vary: origin, accept-encoding, authorization, x-role
< Cache-Control: must-revalidate
< Server: 171123_073418-d8d7cb0 =
< ETag: "zUxy1pv3CQ3IYTFlBg3Z3vYovg3zSw2L"
< x-delay-seconds: 3
< Content-Encoding: gzip
This happens in Firefox, too, although I can't reproduce it at the moment.
The RFC continues, and it looks like the answer linked above is not exact:
unless a cache directive that allows such responses to be stored is present in the response
It looks like the response is cacheable. That's fine, I do want the content to be cached, but I don't want the Authorization header to be served from cache. Is this possible?
Explanation of my problem
My server used to send the Authorization header only when responding to a login request. This used to work fine, problems come with new requirements.
Our site allows users to stay logged in arbitrarily long (we do no sensitive business). We're changing the format of the authorization token and we don't want to force all users to log in again because of this. Therefore, I made the server to send the updated authorization token whenever it sees an obsolete but valid one. So now any response may contain an authorization token, but most of them do not.
The browser cache combining the still valid response with an obsolete authorization token comes in the way.
As a workaround, I made the server send no etag when an authorization token is present. It works, but I'd prefer some cleaner solution.
The quote in the linked answer is misleading because it omitted an important part: "if the cache is shared".
Here's the correct quote (RFC7234 Section 3):
A cache MUST NOT store a response to any request, unless: ... the Authorization header field (see Section 4.2 of [RFC7235]) does not appear in the request, if the cache is shared,
That part of the RFC is basically a summary.
This is the complete rule (RFC7234 Section 3.2) that says essentially the same thing:
A shared cache MUST NOT use a cached response to a request with an Authorization header field (Section 4.2 of [RFC7235]) to satisfy any subsequent request unless a cache directive that allows such responses to be stored is present in the response.
Is a browser cache a shared cache?
This is explained in Introduction section of the RFC:
A private cache, in contrast, is dedicated to a single user; often, they are deployed as a component of a user agent.
That means a browser cache is private cache.
It is not a shared cache, so the above rule does not apply, which means both Chrome and Firefox do their jobs correctly.
Now the solution.
The specification suggests the possibility of a cached response containing Authorization to be reused without the Authorization header.
Unfortunately, it also says that the feature is not widely implemented.
So, the easiest and also the most future-proof solution I can think of is make sure that any response containing Authorization token isn't cached.
For instance, whenever the server sees an obsolete but valid Authorization token, send a new valid one along with Cache-Control: no-store to disallow caching.
Also you must never send Cache-Control: must-revalidate with Authorization header because the must-revalidate directive actually allows the response to be cached, including by shared caches which can cause even more problems in the future.
... unless a cache directive that allows such responses to be stored is present in the response.
In this specification, the following Cache-Control response directives (Section 5.2.2) have such an effect: must-revalidate, public, and s-maxage.
My current solution is to send an authorization header in every response; using a placeholder value of - when no authorization is wanted.
The placeholder value is obviously meaningless and the client knows it and happily ignores it.
This solution is ugly as it adds maybe 20 bytes to every response, but that's still better than occasionally having to resend a whole response content as with the approach mentioned in my question. Moreover, with HTTP/2 it'll be free.
I'm trying to download transactions from a bank account (mine).
Step 1: a form is filled and submitted (POST).
Step 2. subsequent to that, the browser sends a GET
https://accountinfo.corp.xxxxxxx.com.au/AIWeb/ExportAccounts/DownloadExport?OfficeId=201012249&ScheduleId=&FileFormat=CSV-Tran&IsAccountExport=False
The browser receives the file and saves it (default action).
The http response is:
(Status-Line) HTTP/1.1 200 OK
Content-Length 73
Content-Type application/AIUsers
Date Thu, 24 Dec 2015 03:24:22 GMT
p3p CP="NON CUR OTPi OUR NOR UNI"
x-frame-options SAMEORIGIN
x-aspnetmvc-version 1.0
Cache-Control private
Content-Disposition attachment; filename=Accounts_24-12-2015_91456974_T.CSV
I emulate the form submission (with ruby mechanize), wait a few seconds, and agent.get the URL above, as in:
url = "https://accountinfo.corp.westpac.com.au/AIWeb/ExportAccounts/DownloadExport?OfficeId=201012249&ScheduleId=&FileFormat=CSV-Tran&IsAccountExport=False"
download_page = agent.get(url)
The result is incorrect:
<html><body><script>window.parent.location = '/AIWeb/ExportAccounts/ShowErrorMessage?errorCode=3';</script></body></html>
Would appreciate some guidance on how to get the result of the GET.
Regards
There must be something that server expects in headers (and/or) cookies. You can check that using chrome inspector or firebug and add to your request if mechanize supports it (no idea what that is).
I have a question for using Apach JMeter.
Our project, Android apps post json data with "Gzip Compression" to API server.
The Android apps using "Apache HttpClient" and it's "GzipCompressingEntity" class.
For performance testing of API server, I tryed to recording the request by JMeter's Proxy (="HTTP(S) Test Script Recorder").
But the recorded request body was empty.
What I want to do is to send "Gziped HTTP request data" from Apache JMeter to server.
Is there any way for that?
Following is sample of our Android app's request header.
POST /api/test HTTP/1.1
Content-Type: application/json
Transfer-Encoding: chunked
Content-Encoding: gzip
Host: 192.168.11.11:8080
Connection: Keep-Alive
User-Agent: Apache-HttpClient/UNAVAILABLE (java 1.5)
Accept-Encoding: gzip,deflate
RequestBody is Gziped binary data.
What I had done is
Run the "HTTP(S) Test Script Recorder(Proxy Server)" on JMeter.
Set http proxy of Android to use that proxy server.
Execute Android test code of HTTP Client (the post data is compressed by Gzip).
then, that test code finished with failure. (no response from server)
If access directly (without JMeter's Proxy Server), that test succeeded.
Can I send the compressed request data from JMeter just like this?
Add HTTP Header Manager to your test plan and add at least the following headers:
Content-Type: application/json
Content-Encoding: gzip
Add Beanshell PreProcessor as a child of the request which you need to encode and add the following code to it's "Script" area:
import org.apache.commons.io.IOUtils;
import java.util.zip.GZIPOutputStream;
String bodyString = sampler.getArguments().getArgument(0).getValue();
byte [] requestBody = bodyString.getBytes();
ByteArrayOutputStream out = new ByteArrayOutputStream(requestBody.length);
GZIPOutputStream gzip = new GZIPOutputStream(out);
gzip.write(requestBody);
gzip.close();
sampler.getArguments().getArgument(0).setValue(out.toString(0));
It will get your request body, compress it and substitute on the fly so the request will be gzipped.
Encoding is wrong, you need to save it in body like this
sampler.getArguments().getArgument(0).setValue(new String(compressedBody, 0));
My question is pretty simple. Although while searching over, I have not found a simple satisfying answer.
I am using Jquery ajax request to get the data from a server. Server
hosts a rest API that sets the Etag and Cach-control headers to the GET requests. The Server also sets CORS headers to allow the Etag.
The client of the Api is a browser web app. I am using Ajax request to call the Api. Here are the response headers from server after a simple GET request:
Status Code: 200 OK
Access-Control-Allow-Origin: *
Cache-Control: no-transform, max-age=86400
Connection: Keep-Alive
Content-Encoding: gzip
Content-Type: application/json
Date: Sun, 30 Aug 2015 13:23:41 GMT
Etag: "-783704964"
Keep-Alive: timeout=15, max=99
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked
Vary: Accept-Encoding
access-control-allow-headers: X-Requested-With, Content-Type, Etag,Authorization
access-control-allow-methods: GET, POST, DELETE, PUT
All I want to know is:
Do I need to manually collect the Etag from response headers sent from the server and attach an if-no-match header to ajax request?OR the Browser sends it by-default in a conditional get request when it has an 'Etag'
I have done debugging over the network console in the browser and It
seems the browser is doing the conditional GET automatically and
sets the if-no-match header.
if it is right, Suppose, I created a new resource, and then I called the get request. It gives me the past cached data for the first time. But when I reload the page, It gives the updated one. So I am confused that, If the dataset on the server-side has changed and it sends a different Etag, Why doesn't the browser get an updated data set from the server unless I have to reload
Also in case of pagination. Suppose I have a URL /users?next=0. next is a query param where the value for the next changes for every new request. Since each response will get its own 'Etag'. Will the browser store the 'Etag' based on request or it just stores the lastest Etag of the previous get request, irrespective of the URL.
Well, I have somehow figured out the solution myself:
The browser sends the if-no-match header itself when it sees url had the e-tag header on a previous request. Browser saves the e-tag with respect to that URL, so it does not matter how many requests with different URLs happen.
Also, a trick to force the browser to fetch a conditional-get to check the e-tag:
Set the max-age header to the lowest (for me 60s works great)
once the cache expires, thebrowser will send a conditional-get to check if the expired cached resource is valid. If the if-no-match header matches with e-tag. The server sends the response back with 304: Not-Modified header. This means the expired cached resource is valid and can be used.
All,
Every HTTP Request I make to my test REST Service is sent with the method set to GET. Tomcat rejects with a 405 - Unsupported Method. Doesn't matter what I change it to (POST, PUT, etc) Jmeter always sends a GET.
I set up the simplest possible test case by creating a Threadgroup with an HTTP Request Sampler and a View Results Tree. I send a JSON body to the REST Services which just echos back the request along with an ID. Works great with Google's REST Client UI.
Here is the result from the View Results Tree:
Response code: 405
Response message: Method Not Allowed
Response headers:
HTTP/1.1 405 Method Not Allowed
Server: Apache-Coyote/1.1
Allow: POST
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 1045
Date: Fri, 18 Jul 2014 21:39:27 GMT
Here is the RequestMapping from my REST Service
#RequestMapping(method = RequestMethod.POST, consumes = "application/json", produces = "application/json")
Here are some screenshots from my results. I wonder why there are two URI's below the HTTP Request in the tree? Notice the POST request looks correct.
Since the right answer is not provided yet: It's the "Follow Redirects" Option that causes this behavoir under certain circumstances.
see http://www.sqaforums.com/showflat.php?Cat=0&Number=687068&Main=675937
Try to end the 'Path' value of HTTP Request with '/'. It has to remove the GET result in View Results Tree.
I had the same problem. I tried everything also I read this question and all answers before find the thing that worked for me.
Content-Type should be application/json. It can not be text/html.
Set that in HTTP Header Manager. I assume you have set authentication details correctly.
We need to have three things properly set.
Content type which will be application/json
set the endpoint correctly in the path ,which you can see in soup ui
Check the port number on which the api wil get run on [All this u can first check on soupui and then try running the same in jmeter