Varnish 4: Always miss - performance

Question

I'm currently setting up Varnish 4 for my domain.
My Varnish caches all files like charm, except of the DOC-request.
But I can't figure out, why the DOC-request is always a "MISS".
I hope, that you guys can help me.
All details below. If you need more, tell me.
Thank you so much for your help!
Regards,
Hermsi
Varnish-Version:
/ # varnishd -V
varnishd (varnish-4.1.3 revision 5e3b6d2)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2015 Varnish Software AS
curl request:
$ curl -s -D - "https://www.example.com" -o /dev/null
HTTP/1.1 200 OK
Accept-Ranges: bytes
Age: 0
Cache-Control: max-age=0
Content-Type: text/html; charset=UTF-8
Date: Wed, 15 Nov 2017 20:25:14 GMT
Expires: Wed, 15 Nov 2017 20:25:14 GMT
Vary: Accept-Encoding
X-Cache: MISS
X-Cache-Age:
X-Cache-Grace:
X-Cache-Ttl-Remaining:
X-Hits: 0
X-Host: example.com
X-Varnish-Cookie-Debug: Cleaned request cookies:
Transfer-Encoding: chunked
Varnishlog:
* << BeReq >> 786983
- Begin bereq 786982 pass
- Timestamp Start: 1510849297.605099 0.000000 0.000000
- BereqMethod GET
- BereqURL /
- BereqProtocol HTTP/1.1
- BereqHeader Host: www.example.com
- BereqHeader User-Agent: curl/7.26.0
- BereqHeader Accept: */*
- BereqHeader X-Forwarded-Host: www.example.com
- BereqHeader X-Forwarded-Port: 443
- BereqHeader X-Forwarded-Proto: https
- BereqHeader X-Forwarded-Server: lb-proxy-1
- BereqHeader X-Real-Ip: XXX.XX.XXX.XXX
- BereqHeader X-Forwarded-For: XXX.XX.XXX.XXX
- BereqHeader Accept-Encoding: gzip
- BereqHeader X-Device: desktop
- BereqHeader X-Varnish: 786983
- VCL_call BACKEND_FETCH
- VCL_return fetch
- BackendOpen 23 boot.default XXX.XX.XXX.XXX 80 10.42.70.133 39122
- BackendStart XXX.XX.XXX.XXX 80
- Timestamp Bereq: 1510849297.619798 0.014699 0.014699
- Timestamp Beresp: 1510849297.676958 0.071859 0.057160
- BerespProtocol HTTP/1.1
- BerespStatus 200
- BerespReason OK
- BerespHeader Date: Thu, 16 Nov 2017 16:21:37 GMT
- BerespHeader Server: Apache
- BerespHeader Cache-Control: max-age=0
- BerespHeader Expires: Thu, 16 Nov 2017 16:21:37 GMT
- BerespHeader Vary: Accept-Encoding
- BerespHeader Content-Encoding: gzip
- BerespHeader Content-Length: 3252
- BerespHeader Content-Type: text/html; charset=UTF-8
- TTL RFC 0 10 -1 1510849298 1510849298 1510849297 1510849297 0
- VCL_call BACKEND_RESPONSE
- BerespHeader X-Varnish-Cookie-Debug: Cleaned request cookies:
- BerespHeader x-url: /
- BerespHeader x-host: www.example.com
- BerespHeader X-Cache: ZIP
- BereqHeader stale-while-revalidate: max-age=0
- TTL VCL -1 7200 0 1510849298
- TTL VCL 120 7200 0 1510849298
- VCL_return deliver
- Storage malloc Transient
- ObjProtocol HTTP/1.1
- ObjStatus 200
- ObjReason OK
- ObjHeader Date: Thu, 16 Nov 2017 16:21:37 GMT
- ObjHeader Server: Apache
- ObjHeader Cache-Control: max-age=0
- ObjHeader Expires: Thu, 16 Nov 2017 16:21:37 GMT
- ObjHeader Vary: Accept-Encoding
- ObjHeader Content-Encoding: gzip
- ObjHeader Content-Length: 3252
- ObjHeader Content-Type: text/html; charset=UTF-8
- ObjHeader X-Varnish-Cookie-Debug: Cleaned request cookies:
- ObjHeader x-url: /
- ObjHeader x-host: www.example.com
- ObjHeader X-Cache: ZIP
- Fetch_Body 3 length stream
- Gzip u F - 3252 10723 80 80 25946
- BackendReuse 23 boot.default
- Timestamp BerespBody: 1510849297.677148 0.072049 0.000190
- Length 3252
- BereqAcct 328 0 328 247 3252 3499
- End
* << Request >> 786982
- Begin req 786981 rxreq
- Timestamp Start: 1510849297.605008 0.000000 0.000000
- Timestamp Req: 1510849297.605008 0.000000 0.000000
- ReqStart 10.42.115.200 35666
- ReqMethod GET
- ReqURL /
- ReqProtocol HTTP/1.1
- ReqHeader Host: www.example.com
- ReqHeader User-Agent: curl/7.26.0
- ReqHeader Accept: */*
- ReqHeader X-Forwarded-For: XXX.XX.XXX.XXX
- ReqHeader X-Forwarded-Host: www.example.com
- ReqHeader X-Forwarded-Port: 443
- ReqHeader X-Forwarded-Proto: https
- ReqHeader X-Forwarded-Server: lb-proxy-1
- ReqHeader X-Real-Ip: XXX.XX.XXX.XXX
- ReqHeader Accept-Encoding: gzip
- ReqUnset X-Forwarded-For: XXX.XX.XXX.XXX
- ReqHeader X-Forwarded-For: XXX.XX.XXX.XXX, 10.42.115.200
- VCL_call RECV
- ReqUnset X-Forwarded-For: XXX.XX.XXX.XXX, 10.42.115.200
- ReqHeader X-Forwarded-For: XXX.XX.XXX.XXX
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqUnset Accept-Encoding: gzip
- ReqHeader Accept-Encoding: gzip
- ReqHeader X-Device: desktop
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- Debug "XXXX HIT-FOR-PASS"
- HitPass 786980
- VCL_call PASS
- VCL_return fetch
- Link bereq 786983 pass
- Timestamp Fetch: 1510849297.677182 0.072175 0.072175
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Date: Thu, 16 Nov 2017 16:21:37 GMT
- RespHeader Server: Apache
- RespHeader Cache-Control: max-age=0
- RespHeader Expires: Thu, 16 Nov 2017 16:21:37 GMT
- RespHeader Vary: Accept-Encoding
- RespHeader Content-Encoding: gzip
- RespHeader Content-Length: 3252
- RespHeader Content-Type: text/html; charset=UTF-8
- RespHeader X-Varnish-Cookie-Debug: Cleaned request cookies:
- RespHeader x-url: /
- RespHeader x-host: www.example.com
- RespHeader X-Cache: ZIP
- RespHeader X-Varnish: 786982
- RespHeader Age: 0
- RespHeader Via: 1.1 varnish-v4
- VCL_call DELIVER
- RespUnset x-url: /
- RespUnset x-host: www.example.com
- RespUnset X-Cache: ZIP
- RespHeader X-Cache: MISS
- RespHeader X-Hits: 0
- RespHeader X-Host: example.com
- RespUnset Server: Apache
- RespUnset X-Varnish: 786982
- RespUnset Via: 1.1 varnish-v4
- RespHeader X-Cache-TTL-Remaining:
- RespHeader X-Cache-Age:
- RespHeader X-Cache-Grace:
- VCL_return deliver
- Timestamp Process: 1510849297.677222 0.072215 0.000040
- RespHeader Accept-Ranges: bytes
- Debug "RES_MODE 2"
- RespHeader Connection: keep-alive
- Timestamp Resp: 1510849297.677275 0.072268 0.000053
- ReqAcct 290 0 290 446 3252 3698
- End
This is my VCL:
vcl 4.0;
import std;
include "cache_invalidation.vcl";
include "custom_subroutines.vcl";
backend default {
.host = "VARNISH_BACKEND_ADDRESS"; # IP or Hostname of backend
.port = "VARNISH_BACKEND_PORT"; # Port Apache or whatever is listening
.connect_timeout = 5s; # Wait a maximum of 5s for backend connection (Apache, Nginx, etc...)
.first_byte_timeout = 10s; # Wait a maximum of 10s for the first byte to come from your backend
.between_bytes_timeout = 2s; # Wait a maximum of 2s between each bytes sent
.max_connections = 300; # Max parallel connections to our backend
}
sub vcl_recv {
# IP forwarding.
if (req.restarts == 0) {
if (req.http.X-Forwarded-For && !req.http.X-Real-IP) {
set req.http.X-Real-IP = regsub(req.http.X-Forwarded-For, ".*\b(?!10|127|172)(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}).*", "\1");
}else{
set req.http.X-Forwarded-For = req.http.X-Real-IP;
}
}
# Enforce redirect to HTTPS
if (req.http.X-Forwarded-Proto !~ "(?i)https") {
return(synth(720, "https://" + req.http.host + req.url));
}
# Remove the Google Analytics added parameters, useless for our backend
if (req.url ~ "(\?|&)(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=") {
set req.url = regsuball(req.url, "&(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "");
set req.url = regsuball(req.url, "\?(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "?");
set req.url = regsub(req.url, "\?&", "?");
set req.url = regsub(req.url, "\?$", "");
}
# Remove anchor or hash references
if(req.url ~ "\#"){
set req.url = regsub(req.url, "\#.*$", "");
}
# Remove incomplete query string
if(req.url ~ "\?$"){
set req.url = regsub(req.url, "\?$", "");
}
# Remove Google Analytics Cookies, our server does not need it
set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", "");
set req.http.Cookie = regsuball(req.http.Cookie, "_ga=[^;]+(; )?", "");
set req.http.Cookie = regsuball(req.http.Cookie, "_gat=[^;]+(; )?", "");
set req.http.Cookie = regsuball(req.http.Cookie, "utmctr=[^;]+(; )?", "");
set req.http.Cookie = regsuball(req.http.Cookie, "utmcmd.=[^;]+(; )?", "");
set req.http.Cookie = regsuball(req.http.Cookie, "utmccn.=[^;]+(; )?", "");
# Remove DoubleClick offensive cookies
set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(; )?", "");
# Remove the Quant Capital cookies (added by some plugin, all __qca)
set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", "");
# Remove the AddThis cookies
set req.http.Cookie = regsuball(req.http.Cookie, "__atuv.=[^;]+(; )?", "");
# Remove a ";" prefix in the cookie if present
set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*", "");
# Are there cookies left with only spaces or that are empty?
if (req.http.cookie ~ "^\s*$") {
unset req.http.cookie;
}
if (req.url ~ "^[^?]*\.(bmp|bz2|css|doc|eot|flv|gif|gz|ico|jpeg|jpg|js|less|pdf|png|rtf|swf|txt|woff|xml)(\?.*)?$") {
unset req.http.Cookie;
return (hash);
}
if (req.url ~ "^[^?]*\.(mp[34]|rar|tar|tgz|gz|wav|zip|bz2|xz|7z|avi|mov|ogm|mpe?g|mk[av]|webm)(\?.*)?$") {
unset req.http.Cookie;
return (hash);
}
if(req.http.Accept-Encoding)
{
if(req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp[34]|ogg)$")
{
# No point compressing these extensions
unset req.http.Accept-Encoding;
} elsif(req.http.Accept-Encoding ~ "gzip")
{
set req.http.Accept-Encoding = "gzip";
} elsif(req.http.Accept-Encoding ~ "deflate")
{
set req.http.Accept-Encoding = "deflate";
} elsif(req.http.Accept-Encoding ~ "sdch")
{
set req.http.Accept-Encoding = "sdch";
} else
{
# Unknown Algorithm
unset req.http.Accept-Encoding;
}
}
}
# At this point, it is our last chance to make some decision (set|unset|mod) about some header that the backend should see.
sub vcl_backend_fetch {
unset bereq.http.X-Varnish-SESS-ID;
}
sub vcl_hit {
# Custom Headers for debug purposes
set req.http.X-Cache-TTL-Remaining = obj.ttl;
set req.http.X-Cache-Age = obj.keep - obj.ttl;
set req.http.X-Cache-Grace = obj.grace;
}
sub vcl_backend_response {
if (bereq.url ~ "^[^?]*\.(mp[34]|rar|tar|tgz|gz|wav|zip|bz2|xz|7z|avi|mov|ogm|mpe?g|mk[av]|webm)(\?.*)?$") {
unset beresp.http.set-cookie;
set beresp.do_stream = true; # Check memory usage it'll grow in fetch_chunksize blocks (128k by default) if the backend doesn't send a Content-Length header, so only enable it for big objects
set beresp.do_gzip = false; # Don't try to compress it for storage
}
# Do not Gzip image files in Varnish
if (beresp.http.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|mp4|ogg|swf)$") {
set beresp.do_gzip = false;
}
else {
set beresp.do_gzip = true;
set beresp.http.X-Cache = "ZIP";
}
# GZip the cached content if possible
if (beresp.http.content-type ~ "text") {
set beresp.do_gzip = true;
}
# If we are performing a redirection 301 / 302 from the backend and our web server and varnish instances are in the same node, apache mod_rewrite could append it's port
# typically varnish :80 apache/nginx/lighthttp :8080, so the redirect can then often redirect the end-user to a URL on :8080, where it should be :80.
if(beresp.status == 301 || beresp.status == 302) {
set beresp.http.Location = regsub(beresp.http.Location, ":[0-9]+", "");
}
# The example below is explained in detail in load_balancing.vcl as part of one of the strategies that we can take to handle Varnish Grace Mode
set bereq.http.stale-while-revalidate = regsub(beresp.http.Cache-Control, ".*stale-while-revalidate\=([0-9]+).*", "\1");
if(std.real(bereq.http.stale-while-revalidate, 0.0) > 0)
{
set beresp.grace = std.duration(bereq.http.stale-while-revalidate + "s", 10s);
} else {
set beresp.grace = 2h;
}
}
sub vcl_deliver {
# At this point this is our last change to set or unset the response headers before deliver it
if (obj.hits > 0)
{
set resp.http.X-Cache = "HIT";
} else
{
set resp.http.X-Cache = "MISS";
}
set resp.http.X-Hits = obj.hits;
# Remove www of our host in the backend response
set resp.http.X-Host = regsub(req.http.host, "^www\.", "");
# Remove PHP , Apache and Varnish versions
unset resp.http.x-powered-by;
unset resp.http.Server;
unset resp.http.X-Varnish;
unset resp.http.Via;
# Here we are ensuring to add this headers in the response (defined in the hit subroutine) for debug purposes
set resp.http.X-Cache-TTL-Remaining = req.http.X-Cache-TTL-Remaining;
set resp.http.X-Cache-Age = req.http.X-Cache-Age;
set resp.http.X-Cache-Grace = req.http.X-Cache-Grace;
}

Answer 1

The varnish log you are providing does not match the curl request you are doing (wrong user agent for example).
If it is still relevant, in your varnishlog you can see that the backend request contains the headers :
BereqHeader Cache-Control: no-cache
BereqHeader Pragma: no-cache
and the backend response contains the following
BerespHeader Cache-Control: max-age=0
which prevent Varnish from caching the response.
Can you share the varnish log part for the Request (not the BackendRequest) ?
Without it, my guess could be that you always get a miss because the builtin.vcl make your request a pass (you do not do any return (pass) in your conf nonetheless the backend request comes from the pass state
Begin bereq 1048612 pass
It could be triggered by these two rules in the builtin.vcl :
if (req.method != "GET" && req.method != "HEAD") {
/* We only deal with GET and HEAD by default */
return (pass);
}
if (req.http.Authorization || req.http.Cookie) {
/* Not cacheable by default */
return (pass);
}
EDIT
That's weird. You should not have any Cookie using Curl.
Try scoping your cookie handling code and use req.http.Cookie (with an uppercase C) like this :
if (req.http.Cookie) {
# Remove Google Analytics Cookies, our server does not need it
set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", "");
set req.http.Cookie = regsuball(req.http.Cookie, "_ga=[^;]+(; )?", "");
set req.http.Cookie = regsuball(req.http.Cookie, "_gat=[^;]+(; )?", "");
set req.http.Cookie = regsuball(req.http.Cookie, "utmctr=[^;]+(; )?", "");
set req.http.Cookie = regsuball(req.http.Cookie, "utmcmd.=[^;]+(; )?", "");
set req.http.Cookie = regsuball(req.http.Cookie, "utmccn.=[^;]+(; )?", "");
# Remove DoubleClick offensive cookies
set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(; )?", "");
# Remove the Quant Capital cookies (added by some plugin, all __qca)
set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", "");
# Remove the AddThis cookies
set req.http.Cookie = regsuball(req.http.Cookie, "__atuv.=[^;]+(; )?", "");
# Remove a ";" prefix in the cookie if present
set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*", "");
# Are there cookies left with only spaces or that are empty?
if (req.http.Cookie ~ "^\s*$") {
unset req.http.Cookie;
}
}
Another thing here, in your vcl log you can see that Varnish is doing Hit-For-Pass, that is, Varnish is caching the fact that your request is not cached. If you want to truly test, you'll need to purge the content beforehands.
One last thing : your backend server (apache) responds with a Cache-Control : max-age =0 header which prevent caching, you should change that if you want to cache content.