I'm using Next.js custom server, which by default sends strong ETag header on each page GET request. For each request I also manually add Cache-Control header res.setHeader('Cache-Control', 'public, max-age=${60 * 60}, s-maxage=${60 * 60 * 24}, must-revalidate')
Also I'm using Cloudflare's loadbalancing and caching mechanism. According to this documentation https://support.cloudflare.com/hc/en-us/articles/218505467-Using-ETag-Headers-with-Cloudflare I sutisfied each requirement related to strong ETag. And the page rule was configured as following:
page rule configuration screenshot.
First strange behaviour is that on a client in headers I noticed for some reason weak ETag. Also the behaviour of caching is little strange. If I'll open a new site post, I'll be usually faced with cf-cache-status: HIT header, which means that Cloudflare already have prepared cache for this specific post. My expectation is that after page reloading in same tab, the page will be recreated form disk cache, but for some reason it makes a request, result of which is status 304 (Not Modified). BUT, if I'll copy the url and open the same post in new tab, it will be recreated from disk cache (as it should), and also after page reloading it makes request with response 304 (Not Modified).
Direct origin server request-response:
General:
Request URL: ...
Request Method: GET
Status Code: 200 OK
Remote Address: ...
Referrer Policy: strict-origin-when-cross-origin
Response headers:
cache-control: public, max-age=3600, s-maxage=86400, must-revalidate
connection: keep-alive
content-encoding: gzip
content-type: text/html; charset=utf-8
date: Thu, 28 Jan 2021 13:02:42 GMT
etag: "62e33-f1yLCVYVUzVH67UMp/2WOxQqfC8"
transfer-encoding: chunked
vary: Accept-Encoding
x-powered-by: Next.js
Request headers:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate, br
Accept-Language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7,uk;q=0.6
Cache-Control: no-cache
Connection: keep-alive
Cookie: _ga=GA1.1.1009400285.1608315387; _gid=GA1.1.1795438935.1611755271; _gat=1
Host: ip_here
Pragma: no-cache
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36
Cloudflare proxy request-response:
General:
Request URL: ...
Request Method: GET
Status Code: 200
Remote Address: ...
Referrer Policy: strict-origin-when-cross-origin
Response headers:
age: 44448
cache-control: public, max-age=3600, s-maxage=86400, must-revalidate
cf-cache-status: HIT
cf-ray: 618ac665c984351f-KBP
cf-request-id: 07ea9a539d0000351f0faf7000000001
content-encoding: br
content-type: text/html; charset=utf-8
date: Thu, 28 Jan 2021 12:39:13 GMT
etag: W/"62e33-vLyavPBB7axQJnhj/OKdPyUNp+A"
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
nel: {"max_age":604800,"report_to":"cf-nel"}
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=4ASHxxkyb8necnQhGaz8th4W%2FcFEwzrBwYiidFGsHK5ZZJ%2BGEYWbmOlDSC2Uep2TYw8vYUm0DL4WkrdC9xsIK%2F6j0G1QF%2BK5817fQQ%3D%3D"}],"max_age":604800,"group":"cf-nel"}
server: cloudflare
vary: Accept-Encoding
x-powered-by: Next.js
Request headers:
:authority: ...
:method: GET
:path: ...
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
accept-encoding: gzip, deflate, br
accept-language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7,uk;q=0.6
cache-control: max-age=0
cookie: _ga=GA1.2.33365049.1592837735; __cfduid=d58947d60ae6adfce59e944d190e702ae1610627770; _gid=GA1.2.788163791.1611406644; connect.sid=s%3AfLFtBQSejs6Jf0gkdqiRhyzAnu8ImE_J.KukkQj15FMXkfcW6alO6BPUYT8SgPWRqtu2xo2QRe8U; __cflb=0H28urhoa5eNWhDb7XVCL24Hq5fwW5bc37oc693mgWx; _gat=1
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: same-origin
sec-fetch-user: ?1
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36
Why is it happening? In ideal I want to recreate page from disk cache in nearest hour and after it make a revalidation request. And make Cloudflare's cache revalidation each day from origin server.
You can see in headers that ETag is different between origin server and cloudflare
Related
I have a strange behavior on a WordPress site. It is working just fine but several users (on safari) reported seeing error 400. While testing with safari I manage to reproduce the problem with changing the user-agent of the browser. Then I started getting error 400 on each ajax call. I've checked the access.log and all request to admin-ajax.php where served with status 200. But when I check the inspector in Safari, the same ajax request got status 400. And this is happening with each and every single ajax request on every page of the site. I've tried to log out/log in, cleared all cookies, cache and etc. but the error was still there.
The site uses ClouldFlare, so I went there and checked all the security and firewall rules, I didn't found my IP blocked anywhere.
So now the question is how a response with code 200 becomes 400?
Here is also the request and response of ajax call:
Summary
URL: https://www.example.com.com/wp-admin/admin-ajax.php
Status: 400
Source: Local Override
Address: yyy.yyy.yyy.yyy:zzz
Initiator: some-script.min.js:1:2080
Request
:method: POST
:scheme: https
:authority: www.example.com.com
:path: /wp-admin/admin-ajax.php
Accept: application/json, text/plain, */*
Content-Type: application/x-www-form-urlencoded
Origin: https://www.example.com.com
Cookie: some-cookies
Content-Length: 88
Accept-Language: en-us
Host: www.example.com.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15
Referer: https://www.example.com.com/units/main/
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Response
:status: 400
Date: Wed, 27 Jan 2021 12:27:47 GMT
Referrer-Policy: strict-origin-when-cross-origin
X-Content-Type-Options: nosniff
Cache-Control: no-cache, must-revalidate, max-age=0, no-store
X-Frame-Options: SAMEORIGIN
Content-Type: text/html
Access-Control-Allow-Credentials: true
Pragma: no-cache
Set-Cookie: some-cookies
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Access-Control-Allow-Origin: https://www.example.com.com
cf-edge-cache: cache,platform=wordpress
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
nel: {"max_age":604800,"report_to":"cf-nel"}
report-to: {"group":"cf-nel","endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=some-token"}],"max_age":604800}
cf-cache-status: DYNAMIC
cf-request-id: 07e569767900001c377fb67000000001
cf-ray: 618278372fc81c37-SOF
x-robots-tag: noindex
Server: cloudflare
Request Data
MIME Type: application/x-www-form-urlencoded
action: my_ajax_callback
term_id: 1209
page_id
And this is the from the access.log:
xx.xxx.xx.xxx - - [24/Mar/2021:10:16:46 +0000] "POST /wp-admin/admin-ajax.php HTTP/1.0" 200 12203 "https://example.com/some-page/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15"
UPDATE
I have to say that this is happening only on Safari and no other browser. Also I'm running Safari in VM with Mojave, not sure if this is related, but I though it is worthy to mention this.
I have tried to pause the CF service, and this change was visible on all other browsers but Safari continued to server the site over CF (I can see in the response header server that its value was cloudflare while on the other browsers it was nginx). I've flushed the DNS in the terminal and restarted VM a few times but this didn't changed.
I have also disabled all the security and firewall features on CloudFlare after I enabled it again but this also didn't solve the problem. I'm starting to believe that the problem lies some where in Safari and not in CF.
Try turning Cloudflare off, the setting is called "Pause Cloudflare on Site" to isolate it and see if that makes a difference. Maybe you have some goofy modsec rules in place?
I am trying to speed up some ajax calls by facilitating ETags. What I've got at the moment works sometimes (ajax calls that are 304 not modified take 40ms instead of 100ms).
However, what I am noticing is, that chrome apparently only sends the If-None-Match header just for one request but doees never send it on Windows machines. This is quite a pity, as the resource this is about I am loading quite frequently to check for changes. Having ETags work all the time would be really helpful in that case.
Here's what is happening on the wire:
GET /api/book/all/READ HTTP/1.1
Host: something
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36
Accept: */*
Referer: http://something/
Accept-Encoding: gzip, deflate, sdch
Accept-Language: de,en-US;q=0.8,en;q=0.6
Cookie: PHPSESSID=something
and the response:
HTTP/1.1 200 OK
Date: Mon, 04 Apr 2016 06:00:17 GMT
Server: Apache
Expires: Thu, 19 Nov 1981 08:52:00 GMT
ETag: "1"
Cache-Control: "max-age=0, must-revalidate"
Vary: Accept-Encoding
Content-Encoding: gzip
Keep-Alive: timeout=2, max=995
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: application/json;charset=utf8
(as a side note, the ETag value I use refers to the version of the data - each time the data is modified, the version is incremented)
In the next request, on Mac, Chrome will send the appropriate If-None-Match header, and the API will respond with a 304 not modified. However, the second time that resource is being loaded, there will not be any If-None-Match header again.
On Windows, Chrome will just not send the If-None-Match header and therefore the server will respond with the full payload.
The way I would like to have it, is that the browser would always send the If-None-Match header, so that the backend can do a proper job to decide if the cached API response is still fresh.
on mac, this is how the second request (where the If-None-Match header is sent) looks:
GET /api/book/all/READ HTTP/1.1
Host: something
Connection: keep-alive
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36
Accept: */*
Referer: http://something/
Accept-Encoding: gzip, deflate, sdch
Accept-Language: de,en-US;q=0.8,en;q=0.6
Cookie: PHPSESSID=something
If-None-Match: "1"
and the response:
HTTP/1.1 304 Not Modified
Date: Mon, 04 Apr 2016 05:27:19 GMT
Server: Apache
Connection: Keep-Alive
Keep-Alive: timeout=2, max=1000
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Vary: Accept-Encoding
I try to save a backbone model with model.save(), I gave the urlRoot, and I'm waiting for a POST call. I have a REST service on another subdomain which is waiting for the request.
I got these request headers by the model.save().
OPTIONS /user HTTP/1.1
Host: rest.secret.loc
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:20.0) Gecko/20100101 Firefox/20.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Origin: http://members.secret.loc
Access-Control-Request-Method: POST
Access-Control-Request-Headers: content-type
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Sy mentioned that this is a preflight request, so I should respond with the following headers:
HTTP/1.1 200 OK
Cache-Control: max-age=0, no-cache, no-store, must-revalidate
Pragma: no-cache
Content-Type: text/html
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Server: Microsoft-IIS/7.5
X-Powered-By: PHP/5.3.8, ASP.NET
access-control-allow-origin: http://members.secret.loc
Access-Control-Allow-Methods: POST, GET, PUT, DELETE, OPTIONS
Date: Fri, 17 May 2013 01:18:21 GMT
Content-Length: 0
I tried out, but this is not working, the browser does not send another request to the server :S :S :S :S :S
How to fix it?
Is this a backbone bug?
I tried out with Chrome, and in chrome console appears what in firebug not:
The content-type header is not allowed. So I modified the web.config file:
<add name="Access-Control-Allow-Headers" value="Content-Type" />
This fixed the problem.
Btw it still sends the preflight, so I put this code on server side:
if ($_SERVER['REQUEST_METHOD'] == 'OPTIONS')
exit;
So it won't mess up everything...
I'm trying to get Firefox to allow me to make an XmlHttpRequest for data from an application port.
I can see that the initial send triggers an OPTIONS message, and I have trapped this using NetMon, and I can see the HTTP fields...
OPTIONS /any.htm HTTP/1.1
Host: www.mysite.co.uk:10090
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-gb,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Origin: http://www.mysite.co.uk
Access-Control-Request-Method: POST
Access-Control-Request-Headers: content-type
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
I respond to this with an OK to tell FireFox to get a move on and send me the data in the XHR request...
HTTP/1.1 200 OK
Access-Control-Allow-Methods: POST,-GET,-OPTIONS
Access-Control-Allow-Origin: *
Access-Control-Max-Age: 172800000
Server: Kawasu-OrderSub
Date: Wed, 23 Jan 2013 15:00:50 GMT
But the POST is never sent, which I assume means that FireFox is not happy with the 200 OK response. The POST sends data with a Content-Type of "application/json"
I've got FireBug and the Net tab has the XHR button, but this will only show the XHR after the pre-flight check has been passed.
Main Question: How can I see (and debug) what FireFox does not like about the server response?
Much obliged for any help in advance.
You need to include the Access-Control-Allow-Headers: content-type response header. This is because the request has the Access-Control-Request-Headers: content-type header, and this value needs to be included in the response.
I've encountered quite unexpected problem using Tomcat and CAS authorization. I just cannot logout in Firefox. I'm redirected to the logout page, but as soon as I reenter application url in the address bar, it is opened as if I'm logged (and I'm logged actually!).
First I've take a notable amount of attempts to fix something in tomcat config, then I've read logs, but nothing helped me actually before it comes up to my mind to check logout behavior in other browsers.
In other browsers everything work just as expected.
And I'm just stuck and would appreciate if one will give me a hint.
I guess [this question][1] is in some way relative with mine, but, helas, disabling caching on the page which should me logouted doesn't help either.
UPD: Some debug information. Firefox's version is 7.0.1, unfortunately, it is not a public application and I can not provide any url. It looks like response.sendRedirect output is something that Firefox is missing. Here is minimal code that works in any browser except Firefox.
session.invalidate();
response.sendRedirect("https://app:8552/cas/logout");
HEADERS
1st REQUEST - which invalidates session and redirect to CAS logout page
REQUEST HEADERS
Host: dev.service.net
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:7.0.1) Gecko/20100101 Firefox/7.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: keep-alive
Referer: http://dev.service.net/
Cookie: JSESSIONID=53B9469EFE9F130E9694F7406BFAB755
RESPONSE HEADERS
Server: nginx/1.0.4
Date: Thu, 20 Oct 2011 09:20:45 GMT
Content-Type: text/html
Content-Length: 184
Location: https://dev:8552/cas/logout
2nd REQUEST - cas logout page itself
REQUEST HEADERS
Host: dev:8552
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:7.0.1) Gecko/20100101 Firefox/7.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: keep-alive
Referer: http://dev.service.net/
Cookie: JSESSIONID=8A68F008825A0F0D14C6BF803E1332CF; GUEST_LANGUAGE_ID=en_US; COOKIE_SUPPORT=true
RESPONSE HEADERS
Server: Apache-Coyote/1.1
Pragma: no-cache
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: no-cache, no-store
Content-Type: text/html;charset=UTF-8
Content-Language: en-US
Content-Length: 1226
Date: Thu, 20 Oct 2011 15:53:57 GMT
3rd REQUEST - we are retuninig to the page which actually should
redirect us to login page, but it does not.
REQUEST HEADERS
Host: dev.service.net
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:7.0.1) Gecko/20100101 Firefox/7.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: keep-alive
Cookie: JSESSIONID=53B9469EFE9F130E9694F7406BFAB755
RESPONSE HEADERS
Server: Apache-Coyote/1.1
Pragma: no-cache
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: no-cache, no-store
Content-Type: text/html;charset=UTF-8
Content-Language: en-US
Content-Length: 1226
Date: Thu, 20 Oct 2011 13:30:51 GMT
According to the headers, you're maintaining two different sessions on two different hosts. When you request a logout on the first host, you're redirected to the second host (which uses a different session cookie). The session cookie of the second host is in turn indeed invalidated (according to the presence of the Set-Cookie header). But based on the last request, the session has not been recreated on the server side (there is no Set-Cookie header). This means that session.invalidate() before response.sendRedirect() has failed somehow, or that the page is actually requested from the browser cache.
In Firebug you should be able to see if the page is requested from the browser cache by checking the text color of the request in the Net tab. If it's grayed out, then it means that it's been served from the browser cache. For Firefox, the must-revalidate header is actually mandatory next to the no-cache, no-store headers. You need to configure your server to add that entry to the header, or to change/create a Filter for that.
See also:
How to control web page caching, across all browsers?