In Firefox 64 on macOS, when I view one of the disk cache entries under about:cache I see a page that looks something like the code I pasted below. I have deleted most of the long string of code after "security-info" and all but the first three lines of the hex and ascii codes at the end.
How can I restore the content of the file into a human readable format?
I have found quite a few solutions which require tools that only run under Windows, but I'm on macOS. I have also tried to save the hex code (e.g. for the first line: 00000000: 1f 8b 08 00 00 00 00 00 00 03 ed bd fb 72 db 48) into a file with UTF-8 encoding and UNIX linefeeds and .gzip as the file ending and then unpacking it, but all I got was the same hex code again. Do I need to delete the 00000000: at the beginning? Use another file format? Another file ending and/or decoder?
Cache entry information
key: https://www.some-site.com/
fetch count: 2
last fetched: 2018-12-16 20:17:21
last modified: 2018-12-17 06:31:53
expires: Expired Immediately
Data size: 17911 B
Security: This is a secure document.
strongly-framed: 1
security-info: FnhllAKWRHGAlo+ESXykKAAAAAAAAAAAwAAAAAAAAEap ... (and so on)
request-method: POST
request-Accept-Encoding: gzip, deflate, br
request-User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:64.0) Gecko/20100101 Firefox/64.0
response-head: HTTP/1.1 200 OK
Server: Server
Date: Sun, 16 Dec 2018 19:17:14 GMT
Content-Type: text/html; charset=utf-8
Status: 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Request-Id: 2K84AFOOBARX4FNG9D4
ETag: W/"76e8cacfoobar0b8be5"
Cache-Control: max-age=0, private, must-revalidate
X-Runtime: 0.300085
X-Content-Type-Options: nosniff, nosniff
Content-Encoding: gzip
x-amz-rid: 2K84AFOOBARX4FNG9D4
Vary: Accept-Encoding,User-Agent
original-response-headers: Server: Server
Date: Sun, 16 Dec 2018 19:17:14 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Status: 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Request-Id: 2K84AFOOBARX4FNG9D4
ETag: W/"76e8cacfoobar0b8be5"
Cache-Control: max-age=0, private, must-revalidate
X-Runtime: 0.300085
X-Content-Type-Options: nosniff
Content-Encoding: gzip
Set-Cookie: _session_id2=9f298foobarfb208b; path=/; expires=Mon, 17 Dec 2018 01:17:14 -0000; HttpOnly
x-amz-rid: 2K84AFOOBARX4FNG9D4
Vary: Accept-Encoding,User-Agent
net-response-time-onstart: 1266
net-response-time-onstop: 1274
00000000: 1f 8b 08 00 00 00 00 00 00 03 ed bd fb 72 db 48 .............r.H
00000010: 92 2f fc f7 f8 29 d0 9c d9 96 fc b5 78 27 75 b3 ./...)......x'u.
00000020: 25 1f 59 b6 dc ee 76 bb dd 96 3c 9e 6e af 83 01 %.Y...v...<.n...
... (and so on)
You need to extract the hexadecimal from between the colon and the dots or whatever on the right, and convert that back to binary. You will then have, in the case shown, a gzip stream that can be decompressed by gzip. You can use xxd -r -p to convert the hex into binary.
Related
Hi, I have used postman to make a successful (response code 200) POST request, however the cURL code that it postman provides in its "code" section is failing:
This code is generating a 419 error ("unknown error"):
curl -L -X POST 'https://screenpehots' -H 'Content-Type: application/x-www-form-urlencoded' -H 'Cookie: __cfduid=d992d1c6e2c91c59803d953a83ad3dcee1589477309; XSRF-TOKEN=eyJpdiI6IlZOeGpmTVBZYzJnaFpIVXZudGtOM1E9PSIsInZhbHVlIjoiaXJnY2JMaXptTGNCb3U3bVgrUk9xeTJtV3RwSDdwSkN0M1BNSWpNa0tjK1RkUGtiKzlLdG8wbmp0WjZoTmZybyIsIm1hYyI6ImMxYjQyZjk5MTM2NDZmOTNhMGZjZjcyNjhjMTYyNWQ4OTQ4NzY2OWIyMjg1MjI5Mzk0OWZhYjQ2Y2ZhZGM1NjgifQ%3D%3D; laravel_session=eyJpdiI6ImtZdngwK3VXdjFHdjBVcmd3RG1LbGc9PSIsInZhbHVlIjoiN2NaWmJyYStmV3FJMU9LODdjcUZWTGdqV2dtQnBJbTdMaTRNbUZPTkdTXC9rSEVQTGc5dWlwOWZFMVhlTmNiQkEiLCJtYWMiOiIwZGIwMGEzYjA5ZDQ5YWFiMDE3ODJkMjdmZTg5MjUzNzQwZTAwMzUxODRhNTdhYzFiMmQ0MmViYWJlMzM5NDljIn0%3D' --data-urlencode '_token=6c8nSDhYEWVgx8O9pR0tKv70jTcl5zmmyLu3fMoO' --data-urlencode 'device=iphone' --data-urlencode 'url=http://www.amazon.com'
And this code generates a 405 ("method not allowed"):
curl -L -X POST 'https://screenpeek.io/shots' \
-H 'Content-Type: application/json' \
-H 'Cookie: __cfduid=d992d1c6e2c91c59803d953a83ad3dcee1589477309; XSRF-TOKEN=eyJpdiI6IjJCTGdcL0VTaEk1dCtcL1Z2ZXp0Zng5UT09IiwidmFsdWUiOiJOQzdHWFl2UlhcL3JISDNpdDdDMEpGZFVVczROOVVqT0U1cVFrN3RtK2ROdUlFV1U2cEl4MnRTZndtT2hENk5ONCIsIm1hYyI6IjJlNmIxZTQ2ZDE5NzA1OTg1NDEyNGYyNjdiOTEyMzdjM2Y0ZGZmMGY1OTMxMGI4OTVmMTk1ZTk1NjZmZWUzOGQifQ%3D%3D; laravel_session=eyJpdiI6ImI4U1lRamFOYnZqR0ZUU1dlZG1VeFE9PSIsInZhbHVlIjoiaExFSHJ2VjJRNnBPc2RWeWhVOWxMYmd1R2lscVpDdTdOMHhpY3ozNDFYZlFGSzlyZkV4VFhNUlB0aGpCbDFmdyIsIm1hYyI6ImFlNGM4M2Y4ZTBjMGQ3YWQ4MWIwZmQxMTAyNzcxMGFkNTU1ZjRkYTUwYWQ0YmQyNWFkNjgwZTQzY2I3MjMxNDEifQ%3D%3D' \
--data-raw '{
"_token": "6c8nSDhYEWVgx8O9pR0tKv70jTcl5zmmyLu3fMoO",
"device": "iphone",
"url": "http://www.amazon.com"
}'
I have attempted to debug the issue but still haven't managed to work out the cause...
== Info: TLSv1.3 (OUT), TLS handshake, Finished (20):
=> Send SSL data, 52 bytes (0x34)
0000: ...0<.\.#?......D:.a...Tsp...h.%.P.MZP...{..s..*....
== Info: SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
== Info: ALPN, server accepted to use http/1.1
== Info: Server certificate:
== Info: subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
== Info: start date: Feb 4 00:00:00 2020 GMT
== Info: expire date: Oct 9 12:00:00 2020 GMT
== Info: subjectAltName: host "screenpeek.io" matched cert's "screenpeek.io"
== Info: issuer: C=US; ST=CA; L=San Francisco; O=CloudFlare, Inc.; CN=CloudFlare Inc ECC CA-2
== Info: SSL certificate verify ok.
=> Send SSL data, 5 bytes (0x5)
0000: ....Q
=> Send SSL data, 1 bytes (0x1)
0000: .
=> Send header, 739 bytes (0x2e3)
0000: POST /shots HTTP/1.1
0016: Host: screenpeek.io
002b: User-Agent: curl/7.65.3
0044: Accept: */*
0051: Content-Type: application/x-www-form-urlencoded
0082: Cookie: __cfduid=d992d1c6e2c91c59803d953a83ad3dcee1589477309; XS
00c2: RF-TOKEN=eyJpdiI6IlZOeGpmTVBZYzJnaFpIVXZudGtOM1E9PSIsInZhbHVlIjo
0102: iaXJnY2JMaXptTGNCb3U3bVgrUk9xeTJtV3RwSDdwSkN0M1BNSWpNa0tjK1RkUGt
0142: iKzlLdG8wbmp0WjZoTmZybyIsIm1hYyI6ImMxYjQyZjk5MTM2NDZmOTNhMGZjZjc
0182: yNjhjMTYyNWQ4OTQ4NzY2OWIyMjg1MjI5Mzk0OWZhYjQ2Y2ZhZGM1NjgifQ%3D%3
01c2: D; laravel_session=eyJpdiI6ImtZdngwK3VXdjFHdjBVcmd3RG1LbGc9PSIsI
0202: nZhbHVlIjoiN2NaWmJyYStmV3FJMU9LODdjcUZWTGdqV2dtQnBJbTdMaTRNbUZPT
0242: kdTXC9rSEVQTGc5dWlwOWZFMVhlTmNiQkEiLCJtYWMiOiIwZGIwMGEzYjA5ZDQ5Y
0282: WFiMDE3ODJkMjdmZTg5MjUzNzQwZTAwMzUxODRhNTdhYzFiMmQ0MmViYWJlMzM5N
02c2: DljIn0%3D
02cd: Content-Length: 93
02e1:
=> Send data, 93 bytes (0x5d)
0000: _token=6c8nSDhYEWVgx8O9pR0tKv70jTcl5zmmyLu3fMoO&device=iphone&ur
0040: l=http%3A%2F%2Fwww.amazon.com
== Info: upload completely sent off: 93 out of 93 bytes
<= Recv SSL data, 5 bytes (0x5)
0000: .....
<= Recv SSL data, 1 bytes (0x1)
0000: .
== Info: TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
<= Recv SSL data, 230 bytes (0xe6)
0000: ....... .\H....8-......#m.H........W.oC.|...#..eC`>g..'.!..C.{K
0040: .Mc..L.4 .9=....!...e......m.L.&7.v.xN(..k........*.I..|.y.oW..
0080: a..rqHW(w.(.RR{U...$..T...'..2*.4.h.R.AENM..n...%q....X.(g......
00c0: 2ZS.j...Y^.L.R.e._....e.......R.......
== Info: TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
<= Recv SSL data, 230 bytes (0xe6)
0000: ....... ;5.;....8-......#m.H....w.r.8..!../.....X.......&.......
0040: urh.... .,.x$\6Q....7....4?r....l.'.v,..|...o.+.;..0....>L.....9
0080: .6..^....X.~.i......:.......VV|.....P.......6,C....&x.p.*....R..
00c0: ..[8.h4.o.^.R;....v...ZT...7X..O......
== Info: old SSL session ID is stale, removing
<= Recv SSL data, 5 bytes (0x5)
0000: ....j
<= Recv SSL data, 1 bytes (0x1)
0000: .
== Info: Mark bundle as not supporting multiuse
<= Recv header, 20 bytes (0x14)
0000: HTTP/1.1 302 Found
<= Recv header, 37 bytes (0x25)
0000: Date: Thu, 14 May 2020 18:16:08 GMT
<= Recv header, 40 bytes (0x28)
0000: Content-Type: text/html; charset=UTF-8
<= Recv header, 28 bytes (0x1c)
0000: Transfer-Encoding: chunked
<= Recv header, 24 bytes (0x18)
0000: Connection: keep-alive
<= Recv header, 34 bytes (0x22)
0000: Cache-Control: no-cache, private
<= Recv header, 46 bytes (0x2e)
0000: Location: https://screenpeek.io/shots/qkjrm1
<= Recv header, 334 bytes (0x14e)
0000: Set-Cookie: XSRF-TOKEN=eyJpdiI6IlZoRVE3aVZXRGtwSkpSOUJSMFhwMUE9P
0040: SIsInZhbHVlIjoiaW1CVUIyakN4b0R2Y0MyMWRQb2xqRGI3c083ZVFWZmdiZEU0d
0080: k9CdmkrcFBXdkhoSEs4UytuTUxkV3k0Q1I1UCIsIm1hYyI6IjUwNzBmYmY0OWFlN
00c0: DJiNTM1ZmM3NDlhODUxZTQwMTZiMGRiMjNjNmY1NWU5NGQzZWQwNzcwYmQyODVmY
0100: WQ0NzMifQ%3D%3D; expires=Thu, 14-May-2020 20:16:08 GMT; Max-Age=
0140: 7200; path=/
<= Recv header, 29 bytes (0x1d)
0000: X-Frame-Options: SAMEORIGIN
<= Recv header, 33 bytes (0x21)
0000: X-XSS-Protection: 1; mode=block
<= Recv header, 33 bytes (0x21)
0000: X-Content-Type-Options: nosniff
<= Recv header, 26 bytes (0x1a)
0000: CF-Cache-Status: DYNAMIC
<= Recv header, 349 bytes (0x15d)
0000: set-cookie: laravel_session=eyJpdiI6InZBR3ZyK3VBRWtPWkhZR0huOUQx
0040: QlE9PSIsInZhbHVlIjoiVXFMaUlyM3lMeGNrVzgrUHl0YjZYSlYwM0E1TlJyYlJw
0080: Yll2R3F6aGdKQnFnNFFZcE5ZK1M2QVdxbXdUcng3MyIsIm1hYyI6IjcwNmQzNjEx
00c0: ZTg3MzZmOWNhM2M4YjFkODVhMjVjYjYzMTc2N2QyZTllMzczZDIxMzIwZWM2ZThj
0100: YzFiZWM5NTQifQ%3D%3D; expires=Thu, 14-May-2020 20:16:08 GMT; Max
0140: -Age=7200; path=/; httponly
<= Recv header, 100 bytes (0x64)
0000: Expect-CT: max-age=604800, report-uri="https://report-uri.cloudf
0040: lare.com/cdn-cgi/beacon/expect-ct"
<= Recv header, 20 bytes (0x14)
0000: Server: cloudflare
<= Recv header, 30 bytes (0x1e)
0000: CF-RAY: 59369bcb4cfbe664-LHR
<= Recv header, 49 bytes (0x31)
0000: cf-request-id: 02b5ffb30d0000e664ac8a1200000001
<= Recv header, 2 bytes (0x2)
0000:
== Info: Ignoring the response-body
<= Recv data, 135 bytes (0x87)
0000: 17c
0005: <!DOCTYPE html>.<html>. <head>. <meta charset="UTF-8"
0045: />. <meta http-equiv="refresh" content="0;url=https://scr
0085: ee
<= Recv SSL data, 5 bytes (0x5)
0000: .....
<= Recv SSL data, 1 bytes (0x1)
0000: .
<= Recv data, 252 bytes (0xfc)
0000: npeek.io/shots/qkjrm1" />.. <title>Redirecting to https:/
0040: /screenpeek.io/shots/qkjrm1</title>. </head>. <body>.
0080: Redirecting to <a href="https://screenpeek.io/shots/qkjrm1">h
00c0: ttps://screenpeek.io/shots/qkjrm1</a>.. </body>.</html>
<= Recv SSL data, 5 bytes (0x5)
0000: .....
<= Recv SSL data, 1 bytes (0x1)
0000: .
<= Recv data, 5 bytes (0x5)
0000: 0
0003:
== Info: Connection #0 to host screenpeek.io left intact
== Info: Issue another request to this URL: 'https://screenpeek.io/shots/qkjrm1'
== Info: Switch from POST to GET
== Info: Found bundle for host screenpeek.io: 0x7f8d17c16740 [serially]
== Info: Re-using existing connection! (#0) with host screenpeek.io
== Info: Connected to screenpeek.io (104.24.101.9) port 443 (#0)
=> Send SSL data, 5 bytes (0x5)
0000: .....
=> Send SSL data, 1 bytes (0x1)
0000: .
=> Send header, 726 bytes (0x2d6)
0000: POST /shots/qkjrm1 HTTP/1.1
001d: Host: screenpeek.io
0032: User-Agent: curl/7.65.3
004b: Accept: */*
0058: Content-Type: application/x-www-form-urlencoded
0089: Cookie: __cfduid=d992d1c6e2c91c59803d953a83ad3dcee1589477309; XS
00c9: RF-TOKEN=eyJpdiI6IlZOeGpmTVBZYzJnaFpIVXZudGtOM1E9PSIsInZhbHVlIjo
0109: iaXJnY2JMaXptTGNCb3U3bVgrUk9xeTJtV3RwSDdwSkN0M1BNSWpNa0tjK1RkUGt
0149: iKzlLdG8wbmp0WjZoTmZybyIsIm1hYyI6ImMxYjQyZjk5MTM2NDZmOTNhMGZjZjc
0189: yNjhjMTYyNWQ4OTQ4NzY2OWIyMjg1MjI5Mzk0OWZhYjQ2Y2ZhZGM1NjgifQ%3D%3
01c9: D; laravel_session=eyJpdiI6ImtZdngwK3VXdjFHdjBVcmd3RG1LbGc9PSIsI
0209: nZhbHVlIjoiN2NaWmJyYStmV3FJMU9LODdjcUZWTGdqV2dtQnBJbTdMaTRNbUZPT
0249: kdTXC9rSEVQTGc5dWlwOWZFMVhlTmNiQkEiLCJtYWMiOiIwZGIwMGEzYjA5ZDQ5Y
0289: WFiMDE3ODJkMjdmZTg5MjUzNzQwZTAwMzUxODRhNTdhYzFiMmQ0MmViYWJlMzM5N
02c9: DljIn0%3D
02d4:
<= Recv SSL data, 5 bytes (0x5)
0000: ....G
<= Recv SSL data, 1 bytes (0x1)
0000: .
== Info: Mark bundle as not supporting multiuse
<= Recv header, 33 bytes (0x21)
0000: HTTP/1.1 405 Method Not Allowed
<= Recv header, 37 bytes (0x25)
0000: Date: Thu, 14 May 2020 18:16:08 GMT
<= Recv header, 40 bytes (0x28)
0000: Content-Type: text/html; charset=UTF-8
<= Recv header, 28 bytes (0x1c)
0000: Transfer-Encoding: chunked
<= Recv header, 24 bytes (0x18)
0000: Connection: keep-alive
<= Recv header, 18 bytes (0x12)
0000: allow: GET, HEAD
<= Recv header, 34 bytes (0x22)
0000: Cache-Control: no-cache, private
<= Recv header, 26 bytes (0x1a)
0000: CF-Cache-Status: DYNAMIC
<= Recv header, 100 bytes (0x64)
0000: Expect-CT: max-age=604800, report-uri="https://report-uri.cloudf
0040: lare.com/cdn-cgi/beacon/expect-ct"
<= Recv header, 20 bytes (0x14)
0000: Server: cloudflare
<= Recv header, 30 bytes (0x1e)
0000: CF-RAY: 59369bcc4f39e664-LHR
<= Recv header, 49 bytes (0x31)
0000: cf-request-id: 02b5ffb3b10000e664ac8a7200000001
<= Recv header, 2 bytes (0x2)
0000:
<= Recv data, 637 bytes (0x27d)
0000: 276
0005: <!DOCTYPE html>.<html>. <head>. <meta charset="UTF-8"
0045: />. <meta name="robots" content="noindex,nofollow" />.
0085: <style> body { background-color: #fff; color
00c5: : #222; font: 16px/1.5 -apple-system, BlinkMacSystemFont, "Segoe
0105: UI", Roboto, "Helvetica Neue", Arial, sans-serif; margin: 0; }.
0145: .container { margin: 30px; max-width: 600px; }.
0185: h1 { color: #dc3545; font-size: 24px; }</style>.
01c5: </head>. <body>. <div class="contai
0205: ner">. <h1>Whoops, looks like something went
0245: wrong.</h1>. </div>. </body>.</html>
<= Recv SSL data, 5 bytes (0x5)
0000: .....
<= Recv SSL data, 1 bytes (0x1)
0000: .
<= Recv data, 5 bytes (0x5)
0000: 0
0003:
== Info: Connection #0 to host screenpeek.io left intactz
I'm working on a server-side HTTP/2 implementation.
Sometimes clients disconnect with a GOAWAY.COMPRESSION_ERROR in response to the server headers.
According to the Via header, in all of those cases, the client is an intermediary, 'Websense Web Security Gateway v7.6' (looks like a transparent caching solution)
The RFC says that COMPRESSION_ERROR can be sent in 2 cases.
1) A decoding error in a header block MUST be treated as a connection error of type COMPRESSION_ERROR
2) An endpoint receiving HEADERS, PUSH_PROMISE, or CONTINUATION frames needs to reassemble header blocks and perform decompression even if
the frames are to be discarded. A receiver MUST terminate the connection with a connection error of type COMPRESSION_ERROR if it does not decompress a header block
So in my case it's either an error on my side (bad headers encoding) or it's not an error but the client just discarded the response and sent a COMPRESSION_ERROR as specified by RFC.
If it's the latter then I don't need to fix anything.
But if it's the former then it's a bug in our implementation.
I'd like to rule out the bug case.
Below is a sample of such a session terminated with a COMPRESSION_ERROR.
It's hex dumps of data with my interpretation in {}.
S>C stands for server->client direction, C>S - client->server direction.
Can someone review the dump and tell me if it has some obvious issues with S>C frames or my interpretation of the data?
{preface stripped}
========== S>C frame: ============
00000000: 000024040000000000 ..$......
{type=SETTINGS stream=0 flags=NO_FLAGS(0) len=36 data={...}}
00000000: 00020000000000010000100000030000006400047fffffff000600007f380005 .................d...........8..
00000020: 00007ff7 ....
==================================
========== C>S frame: ============
00000000: 000012040000000000 .........
{type=SETTINGS stream=0 flags=NO_FLAGS(0) len=18 data={...}}
00000000: 000200000000000300000064000600008000 ...........d......
==================================
========== S>C frame: ============
00000000: 000000040100000000 .........
{type=SETTINGS stream=0 flags=ACK(1) len=0 data={}}
==================================
========== C>S frame: ============
00000000: 000004080000000000 .........
{type=WINDOW_UPDATE stream=0 flags=NO_FLAGS(0) len=4 data={inc=10485760}}
00000000: 00a00000 ....
==================================
========== C>S frame: ============
00000000: 000000040100000000 .........
{type=SETTINGS stream=0 flags=ACK(1) len=0 data={}}
==================================
========== C>S frame: ============
00000000: 00015c01050000000d ..\......
{type=HEADERS stream=13 flags=END_STREAM|END_HEADERS(5) len=348 data={...}}
00000000: 418aa4b2186cae824952fdea82449360d4cccb02005f603109a637e36a7c475e A....l..IR...D.`....._`1..7.j|G^
00000020: baa687539e352398ac782c75fd1a91cc56075d537d1a91cc563e7ebe58f9fbed ...S.5#..x,u....V.]S}...V>~.X...
00000040: 00177b73dc9d29ad171863c78f0ba0b54875fbd4c7f96a698a92c86b698072ca ..{s..)...c.....Hu....ji...ki.r.
00000060: 6a0c3497e2d4d3120f6d842c03210a8426c1c85f81cb2c4834903cefede170ed j.4......m.,.!..&.._..,H4.<...p.
00000080: 665435fb875c9b1a7b2dfceda35f7a67967e7ea3a19fde8dd02fdbf97d5e2ee9 fT5..\..{-..._zg.~~....../..}^..
000000a0: c97ad5d07f66a281b0dae053fae46aa43f8429a77a8102e0fb5396ae4e35fda9 .z...f.....S..j.?.).z....S..N5..
000000c0: 0d75d05e458f3192c36cbabb2e29fd66c7bf467fa5283752a988a4ea7fed4bd3 .u.^E.1..l...).f..F..(7R......K.
000000e0: d87a4ac3acae05d971e65708195370e51d8661b65d5d973f508d9bd9abfa5242 .zJ.....q.W..Sp...a.]].?P.....RB
00000100: cb40d25fa523b351b82d4b70ddf45abefb4005dffaee6fbed00177bebdcff7da .#._.#.Q.-Kp..Z..#....o...w.....
00000120: 002eefd7b9d6bda7efb4005dcfaf73ad8f37df6800bb7f5ee75b7f2fbed00176 ...........]..s..7.h...^.[./...v
00000140: bf7c99e458d05a9055391635371496d864fa5310d25f03f4a7775773 .|..X.Z.U9.57...d.S.._...wWs
==================================
========== S>C frame: ============
00000000: 00007c01040000000d ..|......
{type=HEADERS stream=13 flags=END_HEADERS(4) len=124 data={...}}
00000000: 3fe11f887684aa6355e76196c361be940bea435d8a08017d4006e32edc684a62 ?...v..cU.a..a....C]...}#....hJb
00000020: d1bf5f87352398ac5754df0f0d83644f3f6c96df3dbf4a09a5328ea50400bea0 .._.5#..WT....dO?l..=.J..2......
00000040: 1ab826ee32da98b46f628cfe5b234fb2566558923ffcff0f1596dd6d5f4a09e5 ..&.2...ob..[#O.VeX.?......m_J..
00000060: 21aec50400bea05ab8cbf700253168df588aa47e561cc581c034f001 !......Z....%1h.X..~V....4..
==================================
========== S>C frame: ============
00000000: 000cd900010000000d .........
{type=DATA stream=13 flags=END_STREAM(1) len=3289 data={...}}
{data stripped}
==================================
========== C>S frame: ============
00000000: 000014070000000000 .........
{type=GOAWAY stream=0 flags=NO_FLAGS(0) len=20 data={...}}
00000000: 0000000000000009000000000000033200000333 ...............2...3
{last_stream_id=0, error=COMPRESSION_ERROR}
==================================
========== S>C frame: ============
00000000: 000008070000000000 .........
{type=GOAWAY stream=0 flags=NO_FLAGS(0) len=8 data={...}}
00000000: 0000000d00000000 ........
==================================
C>S headers decoded:
:path: /img3/2019/1_24/TH4hGo.png HTTP/2.0
:authority: media3.scdn.vn
Accept: image/webp,image/apng,image/*,*/*;q=0.8
Referer: https://www.sendo.vn/?utm_medium=affiliate&utm_source=accesstrade&aff_sid=...
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,vi;q=0.8,zh;q=0.7,zh-CN;q=0.6,zh-HK;q=0.5,zh-TW;q=0.4
Via: Websense Web Security Gateway v7.6
S>C HEADERS frame decoded:
3fe11f
{set dyn table size 4096}
{dyn table size=0/4096}
88
{indexed header ix=8 :status: 200}
{dyn table not changed}
7684aa6355e7
{literal indexed header ix=54 Server: nginx }
{dyn table size=43/4096:
ix=62 off=0 size=43 Server: nginx
}
6196c361be940bea435d8a08017d4006e32edc684a62d1bf
{literal indexed header ix=33 Date: Fri, 19 Apr 2019 01:37:42 GMT }
{dyn table size=108/4096:
ix=62 off=0 size=65 Date: Fri, 19 Apr 2019 01:37:42 GMT
ix=63 off=65 size=43 Server: nginx
}
5f87352398ac5754df
{literal indexed header ix=31 Content-Type: image/png}
{dyn table size=161/4096:
ix=62 off=0 size=53 Content-Type: image/png
ix=63 off=53 size=65 Date: Fri, 19 Apr 2019 01:37:42 GMT
ix=64 off=118 size=43 Server: nginx
}
0f0d83644f3f
{literal not indexed header ix=28 Content-Length: 3289}
{dyn table not changed}
6c96df3dbf4a09a5328ea50400bea01ab826ee32da98b46f
{literal indexed header ix=44 Last-Modified: Thu, 24 Jan 2019 04:25:35 GMT }
{dyn table size=235/4096:
ix=62 off=0 size=74 Last-Modified: Thu, 24 Jan 2019 04:25:35 GMT
ix=63 off=74 size=53 Content-Type: image/png
ix=64 off=127 size=65 Date: Fri, 19 Apr 2019 01:37:42 GMT
ix=65 off=192 size=43 Server: nginx
}
628cfe5b234fb2566558923ffcff
{literal indexed header ix=34 ETag: "5c493e3f-cd9"}
{dyn table size=285/4096:
ix=62 off=0 size=50 ETag: "5c493e3f-cd9"
ix=63 off=50 size=74 Last-Modified: Thu, 24 Jan 2019 04:25:35 GMT
ix=64 off=124 size=53 Content-Type: image/png
ix=65 off=177 size=65 Date: Fri, 19 Apr 2019 01:37:42 GMT
ix=66 off=242 size=43 Server: nginx
}
0f1596dd6d5f4a09e521aec50400bea05ab8cbf700253168df
{literal not indexed header ix=36 Expires: Sun, 28 Apr 2019 14:39:02 GMT}
{dyn table not changed}
588aa47e561cc581c034f001
{literal indexed header ix=24 Cache-Control: max-age=604800}
{dyn table size=344/4096:
ix=62 off=0 size=59 Cache-Control: max-age=604800
ix=63 off=59 size=50 ETag: "5c493e3f-cd9"
ix=64 off=109 size=74 Last-Modified: Thu, 24 Jan 2019 04:25:35 GMT
ix=65 off=183 size=53 Content-Type: image/png
ix=66 off=236 size=65 Date: Fri, 19 Apr 2019 01:37:42 GMT
ix=67 off=301 size=43 Server: nginx
}
The decoded S>C headers:
:status: 200
Server: nginx
Date: Fri, 19 Apr 2019 01:37:42 GMT
Content-Type: image/png
Content-Length: 3289
Last-Modified: Thu, 24 Jan 2019 04:25:35 GMT
ETag: "5c493e3f-cd9"
Expires: Sun, 28 Apr 2019 14:39:02 GMT
Cache-Control: max-age=604800
I added an hpack decoder to h2get here: https://github.com/deweerdt/h2get/pull/18/files and i'm seeing the same results as you are:
$ ./hpack_decoder 3fe11f887684aa6355e76196c361be940bea435d8a08017d4006e32edc684a62d1bf5f87352398ac5754df0f0d83644f3f6c96df3dbf4a09a5328ea50400bea01ab826ee32da98b46f628cfe5b234fb2566558923ffcff0f1596dd6d5f4a09e521aec50400bea05ab8cbf700253168df588aa47e561cc581c034f001
dyn table resize to 4096
[:status] => [200]
[server] => [nginx]
[date] => [Fri, 19 Apr 2019 01:37:42 GMT]
[content-type] => [image/png]
[content-length] => [3289]
[last-modified] => [Thu, 24 Jan 2019 04:25:35 GMT]
[etag] => ["5c493e3f-cd9"]
[expires] => [Sun, 28 Apr 2019 14:39:02 GMT]
[cache-control] => [max-age=604800]
I'm using Spring reactive WebClient for sending requests to a http server. Inorder to view the underlying request & response that's being sent, I enabled debug logging for reactor.ipc.netty package.
The headers for the outgoing requests can be viewed normally.
Tho I'm sending & receiving plain text over http, the log contains the request & responses in the below format (is it hex?)
I'm not sure how to view the logged data in a easy to understand way. Better yet log the request & response in a understandable way
Here is a snippet of the logged data
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 47 45 54 20 2f 53 65 61 72 63 68 5f 47 43 2e 61 |GET /Search_GC.a|
|00000010| 73 70 78 20 48 54 54 50 2f 31 2e 31 0d 0a 75 73 |spx HTTP/1.1..us|
|00000020| 65 72 2d 61 67 65 6e 74 3a 20 52 65 61 63 74 6f |er-agent: Reacto|
|00000030| 72 4e 65 74 74 79 2f 30 2e 37 2e 32 2e 52 45 4c |rNetty/0.7.2.REL|
|00000040| 45 41 53 45 0d 0a 68 6f 73 74 3a 20 63 65 6f 6b |EASE..host: ceok|
|00000050| 61 72 6e 61 74 61 6b 61 2e 6b 61 72 2e 6e 69 63 |arnataka.kar.nic|
|00000060| 2e 69 6e 0d 0a 61 63 63 65 70 74 3a 20 2a 2f 2a |.in..accept: */*|
|00000070| 0d 0a 61 63 63 65 70 74 2d 65 6e 63 6f 64 69 6e |..accept-encodin|
|00000080| 67 3a 20 67 7a 69 70 0d 0a 63 6f 6e 74 65 6e 74 |g: gzip..content|
|00000090| 2d 6c 65 6e 67 74 68 3a 20 30 0d 0a 0d 0a |-length: 0.... |
+--------+-------------------------------------------------+----------------+
Found an unanswered question that must be happening because of the same library: Reading a HttpContent that has a PooledUnsafeDirectByteBuf
Raised an issue here
There seems to an orthodox view that debugging is not required for reactive clients. This is a completely pointless arguments as we use tools like rest client, postman, curl, httpie & others to send request and view response
They changed the reactor.netty.http.client.HttpClient class, after I upgraded to io.projectreactor.netty:reactor-netty-http:1.0.5 the following code is compilable and does what you expect. (I am not sure which is the minimal version, I upgraded from something older, but I guess it's 1.0.0. It is a transitive dependency, I upgraded spring-boot-starter-webflux from 2.3.4.RELEASE to 2.4.4.)
The crucial part is the call of wiretap():
wiretap("reactor.netty.http.client.HttpClient", LogLevel.DEBUG, AdvancedByteBufFormat.TEXTUAL, StandardCharsets.UTF_8)
It logs also the header and body of the request and of the response.
The whole context is this:
package com.example;
import io.netty.handler.logging.LogLevel;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;
import org.springframework.http.HttpMethod;
import org.springframework.http.MediaType;
import org.springframework.http.client.reactive.ReactorClientHttpConnector;
import org.springframework.web.reactive.function.BodyInserters;
import org.springframework.web.reactive.function.client.WebClient;
import reactor.core.publisher.Mono;
import reactor.netty.http.client.HttpClient;
import reactor.netty.transport.logging.AdvancedByteBufFormat;
import java.nio.charset.StandardCharsets;
#Slf4j
class RestClientTest {
private WebClient createWebClient() {
final HttpClient httpClient = HttpClient.create()
.wiretap(HttpClient.class.getCanonicalName(), LogLevel.DEBUG, AdvancedByteBufFormat.TEXTUAL, StandardCharsets.UTF_8);
return WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(httpClient))
.build();
}
#Data
#Builder
#NoArgsConstructor
#AllArgsConstructor
private static class User {
int id;
int userId;
String title;
String body;
}
#Test
void createUsersReactive() {
final WebClient webClient = createWebClient();
final String url = "http://jsonplaceholder.typicode.com/posts";
final Mono<User> userMono = webClient.method(HttpMethod.POST)
.uri(url)
.contentType(MediaType.APPLICATION_JSON_UTF8)
.header("X-My-Header", "MyValue1", "MyValue2")
.body(BodyInserters.fromValue(User.builder().userId(1).title("foo").body("bar").build()))
.retrieve()
.bodyToMono(User.class);
final User user = userMono.block();
log.info("Created user: " + user);
}
}
And the output to log is human readable as you request:
... reactor.netty.http.client.HttpClient : [id:e7d7ed93] REGISTERED
... reactor.netty.http.client.HttpClient : [id:e7d7ed93] CONNECT: jsonplaceholder.typicode.com/<unresolved>:80
... reactor.netty.http.client.HttpClient : [id:e7d7ed93] ACTIVE
... r.netty.http.client.HttpClientConnect : [id:e7d7ed93-1] Handler is being applied: {uri=http://jsonplaceholder.typicode.com/posts, method=POST}
... reactor.netty.http.client.HttpClient : [id:e7d7ed93-1] WRITE: 217B POST /posts HTTP/1.1
user-agent: ReactorNetty/1.0.5
host: jsonplaceholder.typicode.com
accept: */*
Content-Type: application/json;charset=UTF-8
X-My-Header: MyValue1
X-My-Header: MyValue2
content-length: 46
... reactor.netty.http.client.HttpClient : [id:e7d7ed93-1] WRITE: 46B {"id":0,"userId":1,"title":"foo","body":"bar"}
... reactor.netty.http.client.HttpClient : [id:e7d7ed93-1] FLUSH
... reactor.netty.http.client.HttpClient : [id:e7d7ed93-1] READ: 1347B HTTP/1.1 201 Created
Date: Tue, 13 Apr 2021 12:49:33 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 65
X-Powered-By: Express
X-Ratelimit-Limit: 1000
X-Ratelimit-Remaining: 999
X-Ratelimit-Reset: 1618318233
Vary: Origin, X-HTTP-Method-Override, Accept-Encoding
Access-Control-Allow-Credentials: true
Cache-Control: no-cache
Pragma: no-cache
Expires: -1
Access-Control-Expose-Headers: Location
Location: http://jsonplaceholder.typicode.com/posts/101
X-Content-Type-Options: nosniff
Etag: W/"41-0LtsWqhuQ7Zsjlj0tYnOrT/Vw5o"
Via: 1.1 vegur
CF-Cache-Status: DYNAMIC
cf-request-id: 096ce0bd560000736722853000000001
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=laAKjgGcoi8SLu%2F6VX5pQIAksdmj9xi31elC5Ld97eljznKIpYjdkQsittoMJp3lJoQIwOACmj89bKSa%2Ff15gRHRmyasV2Xcl%2FmVjJBJm7ytbWocp39UBd90JwVM"}],"max_age":604800,"group":"cf-nel"}
NEL: {"max_age":604800,"report_to":"cf-nel"}
Server: cloudflare
CF-RAY: 63f4d0a88ed07367-CPH
alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400
Proxy-Connection: Keep-Alive
Connection: Keep-Alive
Set-Cookie: __cfduid=d11c86fbd953f7cf768cf7db0c346f22b1618318173; expires=Thu, 13-May-21 12:49:33 GMT; path=/; domain=.typicode.com; HttpOnly; SameSite=Lax
{
"id": 101,
"userId": 1,
"title": "foo",
"body": "bar"
}
... r.n.http.client.HttpClientOperations : [id:e7d7ed93-1] Received response (auto-read:false) : [Date=Tue, 13 Apr 2021 12:49:33 GMT, Content-Type=application/json; charset=utf-8, X-Powered-By=Express, X-Ratelimit-Limit=1000, X-Ratelimit-Remaining=999, X-Ratelimit-Reset=1618318233, Vary=Origin, X-HTTP-Method-Override, Accept-Encoding, Access-Control-Allow-Credentials=true, Cache-Control=no-cache, Pragma=no-cache, Expires=-1, Access-Control-Expose-Headers=Location, Location=http://jsonplaceholder.typicode.com/posts/101, X-Content-Type-Options=nosniff, Etag=W/"41-0LtsWqhuQ7Zsjlj0tYnOrT/Vw5o", Via=1.1 vegur, CF-Cache-Status=DYNAMIC, cf-request-id=096ce0bd560000736722853000000001, Report-To={"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=laAKjgGcoi8SLu%2F6VX5pQIAksdmj9xi31elC5Ld97eljznKIpYjdkQsittoMJp3lJoQIwOACmj89bKSa%2Ff15gRHRmyasV2Xcl%2FmVjJBJm7ytbWocp39UBd90JwVM"}],"max_age":604800,"group":"cf-nel"}, NEL={"max_age":604800,"report_to":"cf-nel"}, Server=cloudflare, CF-RAY=63f4d0a88ed07367-CPH, alt-svc=h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400, Proxy-Connection=Keep-Alive, Connection=Keep-Alive, Set-Cookie=__cfduid=d11c86fbd953f7cf768cf7db0c346f22b1618318173; expires=Thu, 13-May-21 12:49:33 GMT; path=/; domain=.typicode.com; HttpOnly; SameSite=Lax, content-length=65]
... r.n.http.client.HttpClientOperations : [id:e7d7ed93-1] Received last HTTP packet
... reactor.netty.http.client.HttpClient : [id:e7d7ed93] READ COMPLETE
... com.example.RestClientTest : Created user: RestClientIT.User(id=101, userId=1, title=foo, body=bar)
You can do it with doOnNext(), if you use DataBuffer as your reader:
public Mono<ServerResponse> selectByPost(ServerRequest request) {
Flux<DataBuffer> requestBodyFlux = request.bodyToFlux(DataBuffer.class)
.doOnNext(dataBuffer -> {
if (debug ) {
log.debug(new String(dataBuffer.asByteBuffer().array()));
}
Scannable.from(dataBuffer).tags().forEach(System.out::println);
});
}
This is probably not the best way to do it, it would of course be a nice feature, if netty would provide different ways of logging the payload. Hex does have its benefits, depending on what you need to debug.
Seems like the responding server is returning gzipped content, so it makes sense that you're not able to read it.
If you really want to intercept at the raw HTTP level, ensure your request header does not specify it can accept GZipped content (accept-encoding: gzip).
Another alternative may be to log the request at another layer, when it's already been unzipped from the raw data stream, but not yet processed by your application code - not sure how this would work in Reactive webclient though ;)
I am trying to send the list of IPs as data in curl.
Curl adds null objects in data field when I try to parse the data through a bash variable (Here, $ips). It works fine without using the variable, however, I need to use that since the number of IPs is not constant. Below is exact command/syntax used with output:
mabosvgori-m2:Qualys_Auth_Scan vgori$ ips=`cat PRIVATE_IPs`
mabosvgori-m2:Qualys_Auth_Scan vgori$ echo "$ips"
["XX.XX.XX.XX", "YY.YY.YY.YY", "ZZ.ZZ.ZZ.ZZ",....]
mabosvgori-m2:Qualys_Auth_Scan vgori$ curl -v -H "Content-Type:application/json" -X "POST" -d "$ips" "https://aa.bb.info/TEST/APP/Scan" --trace-ascii /dev/stdout
Warning: --trace-ascii overrides an earlier trace/verbose option
== Info: Trying XX.YY.ZZ.AA...
== Info: Connected to aa.bb.info (XX.YY.ZZ.AA) port 443 (#0)
== Info: TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
== Info: Server certificate: *.bb.info
== Info: Server certificate: DigiCert SHA2 High Assurance Server CA
== Info: Server certificate: DigiCert High Assurance EV Root CA
=> Send header, 178 bytes (0xb2)
0000: POST /TEST/APP/Scan HTTP/1.1
0027: Host: aa.bb.info
003f: User-Agent: curl/7.43.0
0058: Accept: */*
0065: Content-Type:application/json
0084: Content-Length: 6907
009a: Expect: 100-continue
00b0:
<= Recv header, 23 bytes (0x17)
0000: HTTP/1.1 100 Continue
=> Send data, 6907 bytes (0x1afb)
0000: ["XX.XX.XX.XX", "YY.YY.YY.YY", "ZZ.ZZ.ZZ.ZZ",....]
== Info: We are completely uploaded and fine
<= Recv header, 36 bytes (0x24)
0000: HTTP/1.1 500 Internal Server Error
<= Recv header, 46 bytes (0x2e)
0000: Content-Type: application/json;charset=UTF-8
<= Recv header, 37 bytes (0x25)
0000: Date: Tue, 12 Apr 2016 19:05:48 GMT
<= Recv header, 27 bytes (0x1b)
0000: Server: Apache-Coyote/1.1
<= Recv header, 23 bytes (0x17)
0000: Vary: Accept-Encoding
<= Recv header, 69 bytes (0x45)
0000: X-Trace: 1B686AF995169D1B189FCEB49C21AA676B73E2E45F3598FBDC3699E
0040: EB3
<= Recv header, 28 bytes (0x1c)
0000: transfer-encoding: chunked
<= Recv header, 24 bytes (0x18)
0000: Connection: keep-alive
<= Recv header, 2 bytes (0x2)
0000:
<= Recv data, 6163 bytes (0x1813)
0000: 180b
0006: {. "error" : "Cannot invoke method getPlatformId() on null obje
0046: ct",. "stacktrace" : "org.codehaus.groovy.runtime.NullObject.in
0086: vokeMethod(NullObject.java:88)\norg.codehaus.groovy.runtime.call
00c6: site.PogoMetaClassSite.call(PogoMetaClassSite.java:45)\norg.code
0106: haus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteA
0146: rray.java:45)\norg.codehaus.groovy.runtime.callsite.NullCallSite
0186: .call(NullCallSite.java:32)\norg.codehaus.groovy.runtime.callsit
01c6: e.CallSiteArray.defaultCall(CallSiteArray.java:45......
I cannot use --data-urlencode or any other encoding option as the server script doesn't support those.
I really thought it was curl but I figured its cat which was the culprit here.
Using cat PRIVATE_IPs | tr -dc '[:digit:],."[]' resolved this.
Wrong again. I finally figured it's the null-bytes in IP address themselves which give the issue. That is, if the IP address contains something like XX.YY.100.ZZ it will throw this error.
How does one force Firefox to not cache a response? We've tried:
response.setHeader("Cache-Control", "no-cache, no-store, must-revalidate"); // HTTP 1.1.
response.setHeader("Pragma", "no-cache"); // HTTP 1.0.
response.setDateHeader("Expires", 0); // Proxies.
but in about:cache we still see:
Cache entry information
key: http://myhost.my.domain.com:42133/ducc-servlet/timestamp
fetch count: 4
last fetched: 2013-03-04 09:21:40
last modified: 2013-03-04 09:12:53
expires: 1969-12-31 19:00:00
Data size: 24
file on disk: none
Security: This document does not have any security info associated with it.
Client: HTTP
request-method: GET
response-head: HTTP/1.1 200 OK Content-Type: text/html;charset=UTF-8 Cache-Control: no-cache, no-store, must-revalidate
Date: Sun, 03 Mar 2013 14:12:57 GMT
Expires: Sun, 03 Mar 2013 14:12:57 GMT
Content-Length: 24
Server: Jetty(7.4.4.v20110707)
00000000: 32 30 31 33 2e 30 33 2e 30 34 20 30 39 3a 31 32 2013.03.04 09:12
00000010: 3a 35 36 20 4d 6f 6e 0a :56 Mon.
Thanks.
Lou.
Firefox will always (unless it runs out of space) store the data in its cache, so that it can do things like view source, save as, etc by just pulling the data from the cache. It might not use it for normal page loads, of course, if the data was set "no-cache".
If the data is no-store, it will put the cache entry in memory, not on disk, as requested.