Postman POST request working but cURL output failing - bash

Hi, I have used postman to make a successful (response code 200) POST request, however the cURL code that it postman provides in its "code" section is failing:
This code is generating a 419 error ("unknown error"):
curl -L -X POST 'https://screenpehots' -H 'Content-Type: application/x-www-form-urlencoded' -H 'Cookie: __cfduid=d992d1c6e2c91c59803d953a83ad3dcee1589477309; XSRF-TOKEN=eyJpdiI6IlZOeGpmTVBZYzJnaFpIVXZudGtOM1E9PSIsInZhbHVlIjoiaXJnY2JMaXptTGNCb3U3bVgrUk9xeTJtV3RwSDdwSkN0M1BNSWpNa0tjK1RkUGtiKzlLdG8wbmp0WjZoTmZybyIsIm1hYyI6ImMxYjQyZjk5MTM2NDZmOTNhMGZjZjcyNjhjMTYyNWQ4OTQ4NzY2OWIyMjg1MjI5Mzk0OWZhYjQ2Y2ZhZGM1NjgifQ%3D%3D; laravel_session=eyJpdiI6ImtZdngwK3VXdjFHdjBVcmd3RG1LbGc9PSIsInZhbHVlIjoiN2NaWmJyYStmV3FJMU9LODdjcUZWTGdqV2dtQnBJbTdMaTRNbUZPTkdTXC9rSEVQTGc5dWlwOWZFMVhlTmNiQkEiLCJtYWMiOiIwZGIwMGEzYjA5ZDQ5YWFiMDE3ODJkMjdmZTg5MjUzNzQwZTAwMzUxODRhNTdhYzFiMmQ0MmViYWJlMzM5NDljIn0%3D' --data-urlencode '_token=6c8nSDhYEWVgx8O9pR0tKv70jTcl5zmmyLu3fMoO' --data-urlencode 'device=iphone' --data-urlencode 'url=http://www.amazon.com'
And this code generates a 405 ("method not allowed"):
curl -L -X POST 'https://screenpeek.io/shots' \
-H 'Content-Type: application/json' \
-H 'Cookie: __cfduid=d992d1c6e2c91c59803d953a83ad3dcee1589477309; XSRF-TOKEN=eyJpdiI6IjJCTGdcL0VTaEk1dCtcL1Z2ZXp0Zng5UT09IiwidmFsdWUiOiJOQzdHWFl2UlhcL3JISDNpdDdDMEpGZFVVczROOVVqT0U1cVFrN3RtK2ROdUlFV1U2cEl4MnRTZndtT2hENk5ONCIsIm1hYyI6IjJlNmIxZTQ2ZDE5NzA1OTg1NDEyNGYyNjdiOTEyMzdjM2Y0ZGZmMGY1OTMxMGI4OTVmMTk1ZTk1NjZmZWUzOGQifQ%3D%3D; laravel_session=eyJpdiI6ImI4U1lRamFOYnZqR0ZUU1dlZG1VeFE9PSIsInZhbHVlIjoiaExFSHJ2VjJRNnBPc2RWeWhVOWxMYmd1R2lscVpDdTdOMHhpY3ozNDFYZlFGSzlyZkV4VFhNUlB0aGpCbDFmdyIsIm1hYyI6ImFlNGM4M2Y4ZTBjMGQ3YWQ4MWIwZmQxMTAyNzcxMGFkNTU1ZjRkYTUwYWQ0YmQyNWFkNjgwZTQzY2I3MjMxNDEifQ%3D%3D' \
--data-raw '{
"_token": "6c8nSDhYEWVgx8O9pR0tKv70jTcl5zmmyLu3fMoO",
"device": "iphone",
"url": "http://www.amazon.com"
}'
I have attempted to debug the issue but still haven't managed to work out the cause...
== Info: TLSv1.3 (OUT), TLS handshake, Finished (20):
=> Send SSL data, 52 bytes (0x34)
0000: ...0<.\.#?......D:.a...Tsp...h.%.P.MZP...{..s..*....
== Info: SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
== Info: ALPN, server accepted to use http/1.1
== Info: Server certificate:
== Info: subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
== Info: start date: Feb 4 00:00:00 2020 GMT
== Info: expire date: Oct 9 12:00:00 2020 GMT
== Info: subjectAltName: host "screenpeek.io" matched cert's "screenpeek.io"
== Info: issuer: C=US; ST=CA; L=San Francisco; O=CloudFlare, Inc.; CN=CloudFlare Inc ECC CA-2
== Info: SSL certificate verify ok.
=> Send SSL data, 5 bytes (0x5)
0000: ....Q
=> Send SSL data, 1 bytes (0x1)
0000: .
=> Send header, 739 bytes (0x2e3)
0000: POST /shots HTTP/1.1
0016: Host: screenpeek.io
002b: User-Agent: curl/7.65.3
0044: Accept: */*
0051: Content-Type: application/x-www-form-urlencoded
0082: Cookie: __cfduid=d992d1c6e2c91c59803d953a83ad3dcee1589477309; XS
00c2: RF-TOKEN=eyJpdiI6IlZOeGpmTVBZYzJnaFpIVXZudGtOM1E9PSIsInZhbHVlIjo
0102: iaXJnY2JMaXptTGNCb3U3bVgrUk9xeTJtV3RwSDdwSkN0M1BNSWpNa0tjK1RkUGt
0142: iKzlLdG8wbmp0WjZoTmZybyIsIm1hYyI6ImMxYjQyZjk5MTM2NDZmOTNhMGZjZjc
0182: yNjhjMTYyNWQ4OTQ4NzY2OWIyMjg1MjI5Mzk0OWZhYjQ2Y2ZhZGM1NjgifQ%3D%3
01c2: D; laravel_session=eyJpdiI6ImtZdngwK3VXdjFHdjBVcmd3RG1LbGc9PSIsI
0202: nZhbHVlIjoiN2NaWmJyYStmV3FJMU9LODdjcUZWTGdqV2dtQnBJbTdMaTRNbUZPT
0242: kdTXC9rSEVQTGc5dWlwOWZFMVhlTmNiQkEiLCJtYWMiOiIwZGIwMGEzYjA5ZDQ5Y
0282: WFiMDE3ODJkMjdmZTg5MjUzNzQwZTAwMzUxODRhNTdhYzFiMmQ0MmViYWJlMzM5N
02c2: DljIn0%3D
02cd: Content-Length: 93
02e1:
=> Send data, 93 bytes (0x5d)
0000: _token=6c8nSDhYEWVgx8O9pR0tKv70jTcl5zmmyLu3fMoO&device=iphone&ur
0040: l=http%3A%2F%2Fwww.amazon.com
== Info: upload completely sent off: 93 out of 93 bytes
<= Recv SSL data, 5 bytes (0x5)
0000: .....
<= Recv SSL data, 1 bytes (0x1)
0000: .
== Info: TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
<= Recv SSL data, 230 bytes (0xe6)
0000: ....... .\H....8-......#m.H........W.oC.|...#..eC`>g..'.!..C.{K
0040: .Mc..L.4 .9=....!...e......m.L.&7.v.xN(..k........*.I..|.y.oW..
0080: a..rqHW(w.(.RR{U...$..T...'..2*.4.h.R.AENM..n...%q....X.(g......
00c0: 2ZS.j...Y^.L.R.e._....e.......R.......
== Info: TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
<= Recv SSL data, 230 bytes (0xe6)
0000: ....... ;5.;....8-......#m.H....w.r.8..!../.....X.......&.......
0040: urh.... .,.x$\6Q....7....4?r....l.'.v,..|...o.+.;..0....>L.....9
0080: .6..^....X.~.i......:.......VV|.....P.......6,C....&x.p.*....R..
00c0: ..[8.h4.o.^.R;....v...ZT...7X..O......
== Info: old SSL session ID is stale, removing
<= Recv SSL data, 5 bytes (0x5)
0000: ....j
<= Recv SSL data, 1 bytes (0x1)
0000: .
== Info: Mark bundle as not supporting multiuse
<= Recv header, 20 bytes (0x14)
0000: HTTP/1.1 302 Found
<= Recv header, 37 bytes (0x25)
0000: Date: Thu, 14 May 2020 18:16:08 GMT
<= Recv header, 40 bytes (0x28)
0000: Content-Type: text/html; charset=UTF-8
<= Recv header, 28 bytes (0x1c)
0000: Transfer-Encoding: chunked
<= Recv header, 24 bytes (0x18)
0000: Connection: keep-alive
<= Recv header, 34 bytes (0x22)
0000: Cache-Control: no-cache, private
<= Recv header, 46 bytes (0x2e)
0000: Location: https://screenpeek.io/shots/qkjrm1
<= Recv header, 334 bytes (0x14e)
0000: Set-Cookie: XSRF-TOKEN=eyJpdiI6IlZoRVE3aVZXRGtwSkpSOUJSMFhwMUE9P
0040: SIsInZhbHVlIjoiaW1CVUIyakN4b0R2Y0MyMWRQb2xqRGI3c083ZVFWZmdiZEU0d
0080: k9CdmkrcFBXdkhoSEs4UytuTUxkV3k0Q1I1UCIsIm1hYyI6IjUwNzBmYmY0OWFlN
00c0: DJiNTM1ZmM3NDlhODUxZTQwMTZiMGRiMjNjNmY1NWU5NGQzZWQwNzcwYmQyODVmY
0100: WQ0NzMifQ%3D%3D; expires=Thu, 14-May-2020 20:16:08 GMT; Max-Age=
0140: 7200; path=/
<= Recv header, 29 bytes (0x1d)
0000: X-Frame-Options: SAMEORIGIN
<= Recv header, 33 bytes (0x21)
0000: X-XSS-Protection: 1; mode=block
<= Recv header, 33 bytes (0x21)
0000: X-Content-Type-Options: nosniff
<= Recv header, 26 bytes (0x1a)
0000: CF-Cache-Status: DYNAMIC
<= Recv header, 349 bytes (0x15d)
0000: set-cookie: laravel_session=eyJpdiI6InZBR3ZyK3VBRWtPWkhZR0huOUQx
0040: QlE9PSIsInZhbHVlIjoiVXFMaUlyM3lMeGNrVzgrUHl0YjZYSlYwM0E1TlJyYlJw
0080: Yll2R3F6aGdKQnFnNFFZcE5ZK1M2QVdxbXdUcng3MyIsIm1hYyI6IjcwNmQzNjEx
00c0: ZTg3MzZmOWNhM2M4YjFkODVhMjVjYjYzMTc2N2QyZTllMzczZDIxMzIwZWM2ZThj
0100: YzFiZWM5NTQifQ%3D%3D; expires=Thu, 14-May-2020 20:16:08 GMT; Max
0140: -Age=7200; path=/; httponly
<= Recv header, 100 bytes (0x64)
0000: Expect-CT: max-age=604800, report-uri="https://report-uri.cloudf
0040: lare.com/cdn-cgi/beacon/expect-ct"
<= Recv header, 20 bytes (0x14)
0000: Server: cloudflare
<= Recv header, 30 bytes (0x1e)
0000: CF-RAY: 59369bcb4cfbe664-LHR
<= Recv header, 49 bytes (0x31)
0000: cf-request-id: 02b5ffb30d0000e664ac8a1200000001
<= Recv header, 2 bytes (0x2)
0000:
== Info: Ignoring the response-body
<= Recv data, 135 bytes (0x87)
0000: 17c
0005: <!DOCTYPE html>.<html>. <head>. <meta charset="UTF-8"
0045: />. <meta http-equiv="refresh" content="0;url=https://scr
0085: ee
<= Recv SSL data, 5 bytes (0x5)
0000: .....
<= Recv SSL data, 1 bytes (0x1)
0000: .
<= Recv data, 252 bytes (0xfc)
0000: npeek.io/shots/qkjrm1" />.. <title>Redirecting to https:/
0040: /screenpeek.io/shots/qkjrm1</title>. </head>. <body>.
0080: Redirecting to <a href="https://screenpeek.io/shots/qkjrm1">h
00c0: ttps://screenpeek.io/shots/qkjrm1</a>.. </body>.</html>
<= Recv SSL data, 5 bytes (0x5)
0000: .....
<= Recv SSL data, 1 bytes (0x1)
0000: .
<= Recv data, 5 bytes (0x5)
0000: 0
0003:
== Info: Connection #0 to host screenpeek.io left intact
== Info: Issue another request to this URL: 'https://screenpeek.io/shots/qkjrm1'
== Info: Switch from POST to GET
== Info: Found bundle for host screenpeek.io: 0x7f8d17c16740 [serially]
== Info: Re-using existing connection! (#0) with host screenpeek.io
== Info: Connected to screenpeek.io (104.24.101.9) port 443 (#0)
=> Send SSL data, 5 bytes (0x5)
0000: .....
=> Send SSL data, 1 bytes (0x1)
0000: .
=> Send header, 726 bytes (0x2d6)
0000: POST /shots/qkjrm1 HTTP/1.1
001d: Host: screenpeek.io
0032: User-Agent: curl/7.65.3
004b: Accept: */*
0058: Content-Type: application/x-www-form-urlencoded
0089: Cookie: __cfduid=d992d1c6e2c91c59803d953a83ad3dcee1589477309; XS
00c9: RF-TOKEN=eyJpdiI6IlZOeGpmTVBZYzJnaFpIVXZudGtOM1E9PSIsInZhbHVlIjo
0109: iaXJnY2JMaXptTGNCb3U3bVgrUk9xeTJtV3RwSDdwSkN0M1BNSWpNa0tjK1RkUGt
0149: iKzlLdG8wbmp0WjZoTmZybyIsIm1hYyI6ImMxYjQyZjk5MTM2NDZmOTNhMGZjZjc
0189: yNjhjMTYyNWQ4OTQ4NzY2OWIyMjg1MjI5Mzk0OWZhYjQ2Y2ZhZGM1NjgifQ%3D%3
01c9: D; laravel_session=eyJpdiI6ImtZdngwK3VXdjFHdjBVcmd3RG1LbGc9PSIsI
0209: nZhbHVlIjoiN2NaWmJyYStmV3FJMU9LODdjcUZWTGdqV2dtQnBJbTdMaTRNbUZPT
0249: kdTXC9rSEVQTGc5dWlwOWZFMVhlTmNiQkEiLCJtYWMiOiIwZGIwMGEzYjA5ZDQ5Y
0289: WFiMDE3ODJkMjdmZTg5MjUzNzQwZTAwMzUxODRhNTdhYzFiMmQ0MmViYWJlMzM5N
02c9: DljIn0%3D
02d4:
<= Recv SSL data, 5 bytes (0x5)
0000: ....G
<= Recv SSL data, 1 bytes (0x1)
0000: .
== Info: Mark bundle as not supporting multiuse
<= Recv header, 33 bytes (0x21)
0000: HTTP/1.1 405 Method Not Allowed
<= Recv header, 37 bytes (0x25)
0000: Date: Thu, 14 May 2020 18:16:08 GMT
<= Recv header, 40 bytes (0x28)
0000: Content-Type: text/html; charset=UTF-8
<= Recv header, 28 bytes (0x1c)
0000: Transfer-Encoding: chunked
<= Recv header, 24 bytes (0x18)
0000: Connection: keep-alive
<= Recv header, 18 bytes (0x12)
0000: allow: GET, HEAD
<= Recv header, 34 bytes (0x22)
0000: Cache-Control: no-cache, private
<= Recv header, 26 bytes (0x1a)
0000: CF-Cache-Status: DYNAMIC
<= Recv header, 100 bytes (0x64)
0000: Expect-CT: max-age=604800, report-uri="https://report-uri.cloudf
0040: lare.com/cdn-cgi/beacon/expect-ct"
<= Recv header, 20 bytes (0x14)
0000: Server: cloudflare
<= Recv header, 30 bytes (0x1e)
0000: CF-RAY: 59369bcc4f39e664-LHR
<= Recv header, 49 bytes (0x31)
0000: cf-request-id: 02b5ffb3b10000e664ac8a7200000001
<= Recv header, 2 bytes (0x2)
0000:
<= Recv data, 637 bytes (0x27d)
0000: 276
0005: <!DOCTYPE html>.<html>. <head>. <meta charset="UTF-8"
0045: />. <meta name="robots" content="noindex,nofollow" />.
0085: <style> body { background-color: #fff; color
00c5: : #222; font: 16px/1.5 -apple-system, BlinkMacSystemFont, "Segoe
0105: UI", Roboto, "Helvetica Neue", Arial, sans-serif; margin: 0; }.
0145: .container { margin: 30px; max-width: 600px; }.
0185: h1 { color: #dc3545; font-size: 24px; }</style>.
01c5: </head>. <body>. <div class="contai
0205: ner">. <h1>Whoops, looks like something went
0245: wrong.</h1>. </div>. </body>.</html>
<= Recv SSL data, 5 bytes (0x5)
0000: .....
<= Recv SSL data, 1 bytes (0x1)
0000: .
<= Recv data, 5 bytes (0x5)
0000: 0
0003:
== Info: Connection #0 to host screenpeek.io left intactz

Related

Why may Squid give a MISS?

I do a request to my Squid through CloudFront and receive a miss:
< content-type: text/html
< date: Thu, 17 Mar 2022 08:22:46 GMT
< server: Apache/2.4.48 (Ubuntu)
< cache-control: max-age=3153600000, public, only-if-cached, immutable
< last-modified: Thu, 01 Jan 1970 00:00:00 GMT
< x-cache-lookup: MISS from ip-172-31-68-17:3128
< via: 1.1 ip-172-31-68-17 (squid/4.13), 1.1 c929a0b0be95dbd556dd38270accc062.cloudfront.net (CloudFront)
< content-security-policy: default-src * 'unsafe-eval' 'unsafe-inline' 'unsafe-dynamic' data: blob: ws: wss:; img-src http: https: data: mediastream: 'unsafe-eval' 'unsafe-hashes' 'unsafe-inline' blob: ws: wss:; frame-ancestors 'self'; frame-ancestors https://allfilesnews.com https://*.allfilesnews.com
< x-frame-options: SAMEORIGIN
< link: <https://Reb1lBkV3d9kbe7qlaaC7nZDFGNEErQKHQBk51bdA6c.allfilesnews.com/Reb1lBkV3d9kbe7qlaaC7nZDFGNEErQKHQBk51bdA6c/VvaYNP0fJiVQfefYmC>; rel="canonical"
< expires: Thu, 15 Apr 2110 20:00:00 GMT
< vary: Origin
< x-cache: Miss from cloudfront
< x-amz-cf-pop: TLV50-C1
< x-amz-cf-id: t6P2Rr1HqwCmSDmzC4oVh2fzq25ooHzPY4OZPwWras-9tBbMqinr3Q==
You see that max-age is very long and Last-Modified is in deep past. Why then this (repeated, I already queried this URL no more than a few days ago) query to Squid gives a MISS?
Here is my Squid config:
read_timeout 60 minutes
http_port 3128 accel vhost allow-direct
cache_dir ufs /Files/main/cache 120000 16 256
cache_peer 127.0.0.1 parent 9003 0 no-query default
acl all src 0.0.0.0/0.0.0.0
http_access allow all
never_direct allow all
forward_max_tries 5
Note that 120GB of the cache space is nearly not used:
$ df -h /Files/main/
Filesystem Size Used Avail Use% Mounted on
Files/main 125G 187M 125G 1% /Files/main

WHen I test varnish with curl -I http://localhost, I get:

I'm trying to install and configure varnish in mey srever, and I think I've done it, altought when I run the teste i get:
curl -I http://localhost
HTTP/1.1 503 Backend fetch failed
Date: Mon, 26 Aug 2019 17:16:50 GMT
Server: Varnish
Content-Type: text/html; charset=utf-8
Retry-After: 5
X-Varnish: 32770
Age: 0
Via: 1.1 varnish (Varnish/5.2)
Connection: keep-alive
Can any expert help me?
For what i see i get this problem "HTTP/1.1 503 Backend fetch failed", I've already locate more memory but the problem persists...
Best regards,
Leonel Nunes
BereqProtocol HTTP/1.0
BereqHeader Host: www.test.com
BereqHeader X-Forwarded-For: 145.239.78.207
BereqHeader grace: none
BereqHeader X-Varnish: 30
VCL_call BACKEND_FETCH
VCL_return fetch
BackendOpen 29 boot.default 145.239.78.207 8080 145.239.78.207 56942
BackendStart 145.239.78.207 8080
Timestamp Bereq: 1568108230.028067 0.000121 0.000121
BerespProtocol HTTP/1.0
Timestamp Beresp: 1568108230.030388 0.002443 0.002321
BerespProtocol HTTP/1.0
BerespStatus 200
BerespReason OK
BerespHeader Date: Tue, 10 Sep 2019 09:37:10 GMT
BerespHeader Server: Apache
BerespHeader Last-Modified: Fri, 23 Aug 2019 01:46:14 GMT
BerespHeader Accept-Ranges: bytes
BerespHeader Content-Length: 7725
BerespHeader Cache-Control: max-age=31536000
BerespHeader Expires: Wed, 09 Sep 2020 09:37:10 GMT
BerespHeader X-Frame-Options: SAMEORIGIN
BerespHeader Connection: close
BerespHeader Content-Type: image/jpeg
VCL_call BACKEND_RESPONSE
TTL VCL -1 259200 0 1568108230
TTL VCL 120 259200 0 1568108230
VCL_return deliver
Storage malloc Transient
ObjProtocol HTTP/1.0
ObjStatus 200
ObjReason OK
ObjHeader Date: Tue, 10 Sep 2019 09:37:10 GMT
ObjHeader Server: Apache
ObjHeader Last-Modified: Fri, 23 Aug 2019 01:46:14 GMT
ObjHeader Accept-Ranges: bytes
ObjHeader Content-Length: 7725
ObjHeader Cache-Control: max-age=31536000
ObjHeader Expires: Wed, 09 Sep 2020 09:37:10 GMT
ObjHeader X-Frame-Options: SAMEORIGIN
ObjHeader Content-Type: image/jpeg
Fetch_Body 3 length stream
BackendClose 29 boot.default
Timestamp BerespBody: 1568108230.030469 0.002524 0.000082
Length 7725
BereqAcct 192 0 192 309 7725 8034
End
<< Request >> 38
Begin req 37 rxreq
Timestamp Start: 1568108230.176665 0.000000 0.000000
Timestamp Req: 1568108230.176665 0.000000 0.000000
ReqStart 145.239.78.207 40768
ReqMethod GET
ReqURL /pub/static/version1567874318/frontend/Swissup/absolute/pt_PT/images/payments/ww.png
ReqProtocol HTTP/1.0
ReqHeader Host: www.test.com
ReqHeader Connection: close
ReqHeader X-Forwarded-For: 145.239.78.207
VCL_call RECV
ReqHeader grace: none
ReqURL /pub/static/version1567874318/frontend/Swissup/absolute/pt_PT/images/payments/ww.png
VCL_return pass
VCL_call HASH
VCL_return lookup
VCL_call PASS
VCL_return fetch
Link bereq 39 pass
Timestamp Fetch: 1568108230.178857 0.002191 0.002191
RespProtocol HTTP/1.0
RespStatus 200
RespReason OK
RespHeader Date: Tue, 10 Sep 2019 09:37:10 GMT
RespHeader Server: Apache
RespHeader Last-Modified: Tue, 03 Sep 2019 16:11:49 GMT
RespHeader Accept-Ranges: bytes
RespHeader Content-Length: 1003
RespHeader Cache-Control: max-age=31536000, public
RespHeader Expires: Wed, 09 Sep 2020 09:37:10 GMT
RespHeader X-Frame-Options: SAMEORIGIN
RespHeader Content-Type: image/png
RespProtocol HTTP/1.1
RespHeader X-Varnish: 38
RespHeader Age: 0
RespHeader Via: 1.1 varnish (Varnish/5.2)
VCL_call DELIVER
RespUnset Age: 0
VCL_return deliver
Timestamp Process: 1568108230.178877 0.002212 0.000021
RespHeader Connection: close
Timestamp Resp: 1568108230.178912 0.002246 0.000035
ReqAcct 155 0 155 363 1003 1366
End
<< Session >> 37
Begin sess 0 HTTP/1
SessOpen 145.239.78.207 40768 a0 145.239.78.207 80 1568108230.176609 26
Link req 38 rxreq
SessClose REQ_CLOSE 0.002
End
<< BeReq >> 98340
Begin bereq 98339 pass
Timestamp Start: 1568108230.180024 0.000000 0.000000
BereqMethod GET
BereqURL /pub/static/version1567874318/frontend/Swissup/absolute/pt_PT/images/payments/paypal.png
BereqProtocol HTTP/1.0
BereqHeader Host: www.test.com
BereqHeader X-Forwarded-For: 145.239.78.207
BereqHeader grace: none
BereqHeader X-Varnish: 98340
VCL_call BACKEND_FETCH
VCL_return fetch
BackendOpen 29 boot.default 145.239.78.207 8080 145.239.78.207 56974
BackendStart 145.239.78.207 8080
Timestamp Bereq: 1568108230.180132 0.000108 0.000108
BerespProtocol HTTP/1.0
Timestamp Beresp: 1568108230.181992 0.001968 0.001860
BerespProtocol HTTP/1.0
BerespStatus 200
BerespReason OK
BerespHeader Date: Tue, 10 Sep 2019 09:37:10 GMT
BerespHeader Server: Apache
BerespHeader Last-Modified: Tue, 03 Sep 2019 16:11:49 GMT
BerespHeader Accept-Ranges: bytes
BerespHeader Content-Length: 1757
BerespHeader Cache-Control: max-age=31536000, public
BerespHeader Expires: Wed, 09 Sep 2020 09:37:10 GMT
BerespHeader X-Frame-Options: SAMEORIGIN
BerespHeader Connection: close
BerespHeader Content-Type: image/png
VCL_call BACKEND_RESPONSE
TTL VCL -1 259200 0 1568108230
TTL VCL 120 259200 0 1568108230
VCL_return deliver
Storage malloc Transient
ObjProtocol HTTP/1.0
ObjStatus 200
ObjReason OK
ObjHeader Date: Tue, 10 Sep 2019 09:37:10 GMT
ObjHeader Server: Apache
ObjHeader Last-Modified: Tue, 03 Sep 2019 16:11:49 GMT
ObjHeader Accept-Ranges: bytes
ObjHeader Content-Length: 1757
ObjHeader Cache-Control: max-age=31536000, public
ObjHeader Expires: Wed, 09 Sep 2020 09:37:10 GMT
ObjHeader X-Frame-Options: SAMEORIGIN
ObjHeader Content-Type: image/png
Fetch_Body 3 length stream
BackendClose 29 boot.default
Timestamp BerespBody: 1568108230.182080 0.002055 0.000088
Length 1757
BereqAcct 204 0 204 316 1757 2073
End
<< Request >> 98339
Begin req 98338 rxreq
Timestamp Start: 1568108230.179969 0.000000 0.000000
Timestamp Req: 1568108230.179969 0.000000 0.000000
ReqStart 145.239.78.207 40772
ReqMethod GET
ReqURL /pub/static/version1567874318/frontend/Swissup/absolute/pt_PT/images/payments/paypal.png
ReqProtocol HTTP/1.0
ReqHeader Host: www.test.com
ReqHeader Connection: close
ReqHeader X-Forwarded-For: 145.239.78.207
VCL_call RECV
ReqHeader grace: none
ReqURL /pub/static/version1567874318/frontend/Swissup/absolute/pt_PT/images/payments/paypal.png
VCL_return pass
VCL_call HASH
VCL_return lookup
VCL_call PASS
VCL_return fetch
Link bereq 98340 pass
Timestamp Fetch: 1568108230.182061 0.002092 0.002092
RespProtocol HTTP/1.0
RespStatus 200
RespReason OK
RespHeader Date: Tue, 10 Sep 2019 09:37:10 GMT
RespHeader Server: Apache
RespHeader Last-Modified: Tue, 03 Sep 2019 16:11:49 GMT
RespHeader Accept-Ranges: bytes
RespHeader Content-Length: 1757
RespHeader Cache-Control: max-age=31536000, public
RespHeader Expires: Wed, 09 Sep 2020 09:37:10 GMT
RespHeader X-Frame-Options: SAMEORIGIN
RespHeader Content-Type: image/png
RespProtocol HTTP/1.1
RespHeader X-Varnish: 98339
RespHeader Age: 0
RespHeader Via: 1.1 varnish (Varnish/5.2)
VCL_call DELIVER
RespUnset Age: 0
VCL_return deliver
Timestamp Process: 1568108230.182087 0.002118 0.000026
RespHeader Connection: close
Timestamp Resp: 1568108230.182146 0.002177 0.000058
ReqAcct 159 0 159 366 1757 2123
End
Not enough detail for a succinct answer.
To cut the problem in-half, try to browse http address of the backend server?
Another important consideration, Have you properly configured, /etc/varnish/user.vcl ?
Here is a Varnish tutorial... https://www.linode.com/docs/websites/varnish/getting-started-with-varnish-cache/
Finally, find the location of the Varnish log file to get more details.
I will post a complete log:
varnishlog
* << BeReq >> 84
- Begin bereq 83 fetch
- Timestamp Start: 1566844978.393604 0.000000 0.000000
- BereqMethod GET
- BereqURL /index.php/giantpointsadmin/adminhtml_earning_rate/index
- BereqProtocol HTTP/1.1
- BereqHeader Host: www.test.pt
- BereqHeader User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.1 Safari/537.36
- BereqHeader X-Forwarded-For: 51.89.7.90
- BereqHeader grace: none
- BereqHeader Accept-Encoding: gzip
- BereqHeader X-Varnish: 84
- VCL_call BACKEND_FETCH
- VCL_return fetch
- FetchError no backend connection
- Timestamp Beresp: 1566844978.393680 0.000077 0.000077
- Timestamp Error: 1566844978.393687 0.000083 0.000007
- BerespProtocol HTTP/1.1
- BerespStatus 503
- BerespReason Service Unavailable
- BerespReason Backend fetch failed
- BerespHeader Date: Mon, 26 Aug 2019 18:42:58 GMT
- BerespHeader Server: Varnish
- VCL_call BACKEND_ERROR
- BerespHeader Content-Type: text/html; charset=utf-8
- BerespHeader Retry-After: 5
- VCL_return deliver
- Storage malloc Transient
- ObjProtocol HTTP/1.1
- ObjStatus 503
- ObjReason Backend fetch failed
- ObjHeader Date: Mon, 26 Aug 2019 18:42:58 GMT
- ObjHeader Server: Varnish
- ObjHeader Content-Type: text/html; charset=utf-8
- ObjHeader Retry-After: 5
- Length 279
- BereqAcct 0 0 0 0 0 0
- End
<< Request >> 83
Begin req 82 rxreq
Timestamp Start: 1566844978.384062 0.000000 0.000000
Timestamp Req: 1566844978.393264 0.009202 0.009202
ReqStart 51.89.7.90 55715
ReqMethod GET
ReqURL /index.php/giantpointsadmin/adminhtml_earning_rate/index
ReqProtocol HTTP/1.1
ReqHeader Host: www.test.pt
ReqHeader Connection: keep-alive
ReqHeader Accept-Encoding: gzip,deflate
ReqHeader User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.1 Safari/537.36
ReqHeader X-Forwarded-For: 51.89.7.90
VCL_call RECV
ReqHeader grace: none
ReqURL /index.php/giantpointsadmin/adminhtml_earning_rate/index
ReqUnset Accept-Encoding: gzip,deflate
ReqHeader Accept-Encoding: gzip
VCL_return hash
VCL_call HASH
VCL_return lookup
VCL_call MISS
VCL_return fetch
Link bereq 84 fetch
Timestamp Fetch: 1566844978.393832 0.009770 0.000568
RespProtocol HTTP/1.1
RespStatus 503
RespReason Backend fetch failed
RespHeader Date: Mon, 26 Aug 2019 18:42:58 GMT
RespHeader Server: Varnish
RespHeader Content-Type: text/html; charset=utf-8
RespHeader Retry-After: 5
RespHeader X-Varnish: 83
RespHeader Age: 0
RespHeader Via: 1.1 varnish (Varnish/5.2)
VCL_call DELIVER
VCL_return deliver
Timestamp Process: 1566844978.393857 0.009795 0.000025
RespHeader Content-Length: 279
RespHeader Connection: keep-alive
Timestamp Resp: 1566844978.393929 0.009868 0.000072
ReqAcct 266 0 266 247 279 526
End
<< Session >> 82
Begin sess 0 HTTP/1
SessOpen 51.89.7.90 55715 a0 145.239.78.207 80 1566844978.383924 25
Link req 83 rxreq
SessClose REM_CLOSE 0.020
End
<< BeReq >> 32855
Begin bereq 32854 pass
Timestamp Start: 1566844978.423125 0.000000 0.000000
BereqMethod POST
BereqURL /index.php/giantpointsadmin/adminhtml_earning_rate/index/
BereqProtocol HTTP/1.1
BereqHeader Host: www.test.pt
BereqHeader Accept-Encoding: gzip,deflate
BereqHeader Content-Type: multipart/form-data; boundary=----------------7O12ZqAnhn621bBI
BereqHeader Content-Length: 326
BereqHeader User-Agent: Opera/9.80 (Windows NT 6.0); U) Presto/2.12.388 Version/12.16
BereqHeader Cookie: adminhtml=86g8r80a2uhrf66b95f2lnslo0
BereqHeader X-Forwarded-For: 51.89.7.90
BereqHeader X-Varnish: 32855
VCL_call BACKEND_FETCH
VCL_return fetch
FetchError no backend connection
Timestamp Beresp: 1566844978.423157 0.000033 0.000033
Timestamp Error: 1566844978.423165 0.000040 0.000008
BerespProtocol HTTP/1.1
BerespStatus 503
BerespReason Service Unavailable
BerespReason Backend fetch failed
BerespHeader Date: Mon, 26 Aug 2019 18:42:58 GMT
BerespHeader Server: Varnish
VCL_call BACKEND_ERROR
BerespHeader Content-Type: text/html; charset=utf-8
BerespHeader Retry-After: 5
VCL_return deliver
Storage malloc Transient
ObjProtocol HTTP/1.1
ObjStatus 503
ObjReason Backend fetch failed
ObjHeader Date: Mon, 26 Aug 2019 18:42:58 GMT
ObjHeader Server: Varnish
ObjHeader Content-Type: text/html; charset=utf-8
ObjHeader Retry-After: 5
Length 282
BereqAcct 0 0 0 0 0 0
End
<< Request >> 32854
Begin req 32853 rxreq
Timestamp Start: 1566844978.413495 0.000000 0.000000
Timestamp Req: 1566844978.422953 0.009458 0.009458
ReqStart 51.89.7.90 55717
ReqMethod POST
ReqURL /index.php/giantpointsadmin/adminhtml_earning_rate/index/
ReqProtocol HTTP/1.1
ReqHeader Host: www.test.pt
ReqHeader Connection: keep-alive
ReqHeader Accept-Encoding: gzip,deflate
ReqHeader Content-Type: multipart/form-data; boundary=----------------7O12ZqAnhn621bBI
ReqHeader Content-Length: 326
ReqHeader User-Agent: Opera/9.80 (Windows NT 6.0); U) Presto/2.12.388 Version/12.16
ReqHeader Cookie: adminhtml=86g8r80a2uhrf66b95f2lnslo0
ReqHeader X-Forwarded-For: 51.89.7.90
VCL_call RECV
VCL_return pass
VCL_call HASH
VCL_return lookup
VCL_call PASS
VCL_return fetch
Link bereq 32855 pass
Timestamp Fetch: 1566844978.423294 0.009799 0.000341
Storage malloc Transient
Timestamp ReqBody: 1566844978.423314 0.009819 0.000020
RespProtocol HTTP/1.1
RespStatus 503
RespReason Backend fetch failed
RespHeader Date: Mon, 26 Aug 2019 18:42:58 GMT
RespHeader Server: Varnish
RespHeader Content-Type: text/html; charset=utf-8
RespHeader Retry-After: 5
RespHeader X-Varnish: 32854
RespHeader Age: 0
RespHeader Via: 1.1 varnish (Varnish/5.2)
VCL_call DELIVER
VCL_return deliver
Timestamp Process: 1566844978.423336 0.009841 0.000022
RespHeader Content-Length: 282
RespHeader Connection: keep-alive
Timestamp Resp: 1566844978.423392 0.009897 0.000056
ReqAcct 374 326 700 250 282 532
End
<< Session >> 32853
Begin sess 0 HTTP/1
SessOpen 51.89.7.90 55717 a0 145.239.78.207 80 1566844978.413439 24
Link req 32854 rxreq
SessClose REM_CLOSE 0.020
End
Best regards,
Leonel Nunes

Decoding Firefox browser cache entry from hex format

In Firefox 64 on macOS, when I view one of the disk cache entries under about:cache I see a page that looks something like the code I pasted below. I have deleted most of the long string of code after "security-info" and all but the first three lines of the hex and ascii codes at the end.
How can I restore the content of the file into a human readable format?
I have found quite a few solutions which require tools that only run under Windows, but I'm on macOS. I have also tried to save the hex code (e.g. for the first line: 00000000: 1f 8b 08 00 00 00 00 00 00 03 ed bd fb 72 db 48) into a file with UTF-8 encoding and UNIX linefeeds and .gzip as the file ending and then unpacking it, but all I got was the same hex code again. Do I need to delete the 00000000: at the beginning? Use another file format? Another file ending and/or decoder?
Cache entry information
key: https://www.some-site.com/
fetch count: 2
last fetched: 2018-12-16 20:17:21
last modified: 2018-12-17 06:31:53
expires: Expired Immediately
Data size: 17911 B
Security: This is a secure document.
strongly-framed: 1
security-info: FnhllAKWRHGAlo+ESXykKAAAAAAAAAAAwAAAAAAAAEap ... (and so on)
request-method: POST
request-Accept-Encoding: gzip, deflate, br
request-User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:64.0) Gecko/20100101 Firefox/64.0
response-head: HTTP/1.1 200 OK
Server: Server
Date: Sun, 16 Dec 2018 19:17:14 GMT
Content-Type: text/html; charset=utf-8
Status: 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Request-Id: 2K84AFOOBARX4FNG9D4
ETag: W/"76e8cacfoobar0b8be5"
Cache-Control: max-age=0, private, must-revalidate
X-Runtime: 0.300085
X-Content-Type-Options: nosniff, nosniff
Content-Encoding: gzip
x-amz-rid: 2K84AFOOBARX4FNG9D4
Vary: Accept-Encoding,User-Agent
original-response-headers: Server: Server
Date: Sun, 16 Dec 2018 19:17:14 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Status: 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Request-Id: 2K84AFOOBARX4FNG9D4
ETag: W/"76e8cacfoobar0b8be5"
Cache-Control: max-age=0, private, must-revalidate
X-Runtime: 0.300085
X-Content-Type-Options: nosniff
Content-Encoding: gzip
Set-Cookie: _session_id2=9f298foobarfb208b; path=/; expires=Mon, 17 Dec 2018 01:17:14 -0000; HttpOnly
x-amz-rid: 2K84AFOOBARX4FNG9D4
Vary: Accept-Encoding,User-Agent
net-response-time-onstart: 1266
net-response-time-onstop: 1274
00000000: 1f 8b 08 00 00 00 00 00 00 03 ed bd fb 72 db 48 .............r.H
00000010: 92 2f fc f7 f8 29 d0 9c d9 96 fc b5 78 27 75 b3 ./...)......x'u.
00000020: 25 1f 59 b6 dc ee 76 bb dd 96 3c 9e 6e af 83 01 %.Y...v...<.n...
... (and so on)
You need to extract the hexadecimal from between the colon and the dots or whatever on the right, and convert that back to binary. You will then have, in the case shown, a gzip stream that can be decompressed by gzip. You can use xxd -r -p to convert the hex into binary.

curl sending null objects in data field when parsing through a bash variable

I am trying to send the list of IPs as data in curl.
Curl adds null objects in data field when I try to parse the data through a bash variable (Here, $ips). It works fine without using the variable, however, I need to use that since the number of IPs is not constant. Below is exact command/syntax used with output:
mabosvgori-m2:Qualys_Auth_Scan vgori$ ips=`cat PRIVATE_IPs`
mabosvgori-m2:Qualys_Auth_Scan vgori$ echo "$ips"
["XX.XX.XX.XX", "YY.YY.YY.YY", "ZZ.ZZ.ZZ.ZZ",....]
mabosvgori-m2:Qualys_Auth_Scan vgori$ curl -v -H "Content-Type:application/json" -X "POST" -d "$ips" "https://aa.bb.info/TEST/APP/Scan" --trace-ascii /dev/stdout
Warning: --trace-ascii overrides an earlier trace/verbose option
== Info: Trying XX.YY.ZZ.AA...
== Info: Connected to aa.bb.info (XX.YY.ZZ.AA) port 443 (#0)
== Info: TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
== Info: Server certificate: *.bb.info
== Info: Server certificate: DigiCert SHA2 High Assurance Server CA
== Info: Server certificate: DigiCert High Assurance EV Root CA
=> Send header, 178 bytes (0xb2)
0000: POST /TEST/APP/Scan HTTP/1.1
0027: Host: aa.bb.info
003f: User-Agent: curl/7.43.0
0058: Accept: */*
0065: Content-Type:application/json
0084: Content-Length: 6907
009a: Expect: 100-continue
00b0:
<= Recv header, 23 bytes (0x17)
0000: HTTP/1.1 100 Continue
=> Send data, 6907 bytes (0x1afb)
0000: ["XX.XX.XX.XX", "YY.YY.YY.YY", "ZZ.ZZ.ZZ.ZZ",....]
== Info: We are completely uploaded and fine
<= Recv header, 36 bytes (0x24)
0000: HTTP/1.1 500 Internal Server Error
<= Recv header, 46 bytes (0x2e)
0000: Content-Type: application/json;charset=UTF-8
<= Recv header, 37 bytes (0x25)
0000: Date: Tue, 12 Apr 2016 19:05:48 GMT
<= Recv header, 27 bytes (0x1b)
0000: Server: Apache-Coyote/1.1
<= Recv header, 23 bytes (0x17)
0000: Vary: Accept-Encoding
<= Recv header, 69 bytes (0x45)
0000: X-Trace: 1B686AF995169D1B189FCEB49C21AA676B73E2E45F3598FBDC3699E
0040: EB3
<= Recv header, 28 bytes (0x1c)
0000: transfer-encoding: chunked
<= Recv header, 24 bytes (0x18)
0000: Connection: keep-alive
<= Recv header, 2 bytes (0x2)
0000:
<= Recv data, 6163 bytes (0x1813)
0000: 180b
0006: {. "error" : "Cannot invoke method getPlatformId() on null obje
0046: ct",. "stacktrace" : "org.codehaus.groovy.runtime.NullObject.in
0086: vokeMethod(NullObject.java:88)\norg.codehaus.groovy.runtime.call
00c6: site.PogoMetaClassSite.call(PogoMetaClassSite.java:45)\norg.code
0106: haus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteA
0146: rray.java:45)\norg.codehaus.groovy.runtime.callsite.NullCallSite
0186: .call(NullCallSite.java:32)\norg.codehaus.groovy.runtime.callsit
01c6: e.CallSiteArray.defaultCall(CallSiteArray.java:45......
I cannot use --data-urlencode or any other encoding option as the server script doesn't support those.
I really thought it was curl but I figured its cat which was the culprit here.
Using cat PRIVATE_IPs | tr -dc '[:digit:],."[]' resolved this.
Wrong again. I finally figured it's the null-bytes in IP address themselves which give the issue. That is, if the IP address contains something like XX.YY.100.ZZ it will throw this error.

firefox no-cache not honored?

How does one force Firefox to not cache a response? We've tried:
response.setHeader("Cache-Control", "no-cache, no-store, must-revalidate"); // HTTP 1.1.
response.setHeader("Pragma", "no-cache"); // HTTP 1.0.
response.setDateHeader("Expires", 0); // Proxies.
but in about:cache we still see:
Cache entry information
key: http://myhost.my.domain.com:42133/ducc-servlet/timestamp
fetch count: 4
last fetched: 2013-03-04 09:21:40
last modified: 2013-03-04 09:12:53
expires: 1969-12-31 19:00:00
Data size: 24
file on disk: none
Security: This document does not have any security info associated with it.
Client: HTTP
request-method: GET
response-head: HTTP/1.1 200 OK Content-Type: text/html;charset=UTF-8 Cache-Control: no-cache, no-store, must-revalidate
Date: Sun, 03 Mar 2013 14:12:57 GMT
Expires: Sun, 03 Mar 2013 14:12:57 GMT
Content-Length: 24
Server: Jetty(7.4.4.v20110707)
00000000: 32 30 31 33 2e 30 33 2e 30 34 20 30 39 3a 31 32 2013.03.04 09:12
00000010: 3a 35 36 20 4d 6f 6e 0a :56 Mon.
Thanks.
Lou.
Firefox will always (unless it runs out of space) store the data in its cache, so that it can do things like view source, save as, etc by just pulling the data from the cache. It might not use it for normal page loads, of course, if the data was set "no-cache".
If the data is no-store, it will put the cache entry in memory, not on disk, as requested.

Resources