I have found this little script in PHP that send a simple request to twitter for update your status, I have tried this: http://pratham.name/twitter-php-script-without-curl.html, and it work. Now, I want send this request with netcat, but this doesn't work, why?
I send request in this way:
echo -e $head | nc twitter.com 80
The $head are my header that I have tried with also PHP, so it are right.
Anyone know how I can make this? Thanks to all.
edit.
head="POST http://twitter.com/statuses/update.json HTTP/1.1\r\n
Host: twitter.com\r\n
Authorization: Basic myname:passwordinbs64\r\n
Content-type: application/x-www-form-urlencoded\r\n
Content-length: 10\r\n
Connection: Close\r\n\r\n
status=mymessage";
echo -e $head | nc twitter.com 80
The data you send is not correct. Here's the command I'm executing:
$ head="POST /statuses/update.json HTTP/1.1\r\n
Host: twitter.com\r\n
Authorization: Basic myname:passwordinbs64\r\n
Content-type: application/x-www-form-urlencoded\r\n
Content-length: 10\r\n
Connection: Close\r\n\r\n
status=mymessage"; echo $head | nc twitter.com 80
HTTP/1.1 400 Bad Request
Date: Tue, 29 Jun 2010 10:19:38 GMT
Server: Apache
Vary: Accept-Encoding
Content-Length: 226
Connection: close
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
</body></html>
I'm running a tcpdump in parallel and here is what it shows me:
0x0020: 5018 ffff 671f 0000 504f 5354 2068 7474 P...g...POST.htt
0x0030: 703a 2f2f 7477 6974 7465 722e 636f 6d2f p://twitter.com/
0x0040: 7374 6174 7573 6573 2f75 7064 6174 652e statuses/update.
0x0050: 6a73 6f6e 2048 5454 502f 312e 310d 0a20 json.HTTP/1.1...
0x0060: 486f 7374 3a20 7477 6974 7465 722e 636f Host:.twitter.co
0x0070: 6d0d 0a20 4175 7468 6f72 697a 6174 696f m...Authorizatio
0x0080: 6e3a 2042 6173 6963 206d 796e 616d 653a n:.Basic.myname:
0x0090: 7061 7373 776f 7264 696e 6273 3634 0d0a passwordinbs64..
0x00a0: 2043 6f6e 7465 6e74 2d74 7970 653a 2061 .Content-type:.a
0x00b0: 7070 6c69 6361 7469 6f6e 2f78 2d77 7777 pplication/x-www
0x00c0: 2d66 6f72 6d2d 7572 6c65 6e63 6f64 6564 -form-urlencoded
0x00d0: 0d0a 2043 6f6e 7465 6e74 2d6c 656e 6774 ...Content-lengt
0x00e0: 683a 2031 300d 0a20 436f 6e6e 6563 7469 h:.10...Connecti
0x00f0: 6f6e 3a20 436c 6f73 650d 0a0d 0a20 7374 on:.Close.....st
0x0100: 6174 7573 3d6d 796d 6573 7361 6765 0a atus=mymessage.
Do you see the mistake? At each line break are three characters, but there should be only two. This happens because you have a real line break in your variable definition that becomes a space in the shell. E.g.
$ head="line1
line2" ; echo $head
line1 line2
Try writing it all in one line instead.
$ head="POST /statuses/update.json HTTP/1.1\r\nHost: twitter.com\r\nAuthorization: Basic myname:passwordinbs64\r\nContent-type: application/x-www-form-urlencoded\r\nContent-length: 10\r\nConnection: Close\r\n\r\nstatus=mymessage" ; echo $head | nc twitter.com 80
HTTP/1.1 401 Unauthorized
Date: Tue, 29 Jun 2010 10:25:43 GMT
Server: hi
Status: 401 Unauthorized
X-Runtime: 0.00577
Content-Type: application/json; charset=utf-8
Content-Length: 63
Cache-Control: no-cache, max-age=300
Set-Cookie: k=194.145.236.196.1277807143945525; path=/; expires=Tue, 06-Jul-10 10:25:43 GMT; domain=.twitter.com
Set-Cookie: guest_id=1277807143950191; path=/; expires=Thu, 29 Jul 2010 10:25:43 GMT
Set-Cookie: _twitter_sess=BAh7CDoPY3JlYXRlZF9hdGwrCBOsPYMpAToHaWQiJWQzZDkxNTg2MTY3OTFk%250AMzg5OTUyNWFhZGZlZDY0YjJmIgpmbGFzaElDOidBY3Rpb25Db250cm9sbGVy%250AOjpGbGFzaDo6Rmxhc2hIYXNoewAGOgpAdXNlZHsA--e95d14c755b85ff19606a0bc907c8b7a63d1b508; domain=.twitter.com; path=/
Expires: Tue, 29 Jun 2010 10:30:43 GMT
Vary: Accept-Encoding
Connection: close
{"errors":[{"code":32,"message":"Could not authenticate you"}]}
If you don't put quotes around "$head" then your formatting won't be as you (or Twitter) expect it. Also, since your variable contains newlines/carriage returns, it will probably help to suppress echo from supplying its own.
echo -ne "$head" | nc twitter.com 80
Related
If I run the following:
sed 's/$/\r/g' <<EOF | netcat "www.google.com" 80
GET / HTTP/1.1
Host:www.google.com
Connection: close
EOF
I get the usual headers, followed by a hexadecimal number before the response body I haven't seen before (in this case, 5172, although another example is 51C8):
HTTP/1.1 200 OK
Date: Thu, 09 Sep 2021 20:03:53 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Server: gws
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
Set-Cookie: ...
Accept-Ranges: none
Vary: Accept-Encoding
Connection: close
Transfer-Encoding: chunked
5172
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en-GB"><head>...
In presumed this was content length, although the response body length did not match this by a large order.
What is this number?
I'm working on a server-side HTTP/2 implementation.
Sometimes clients disconnect with a GOAWAY.COMPRESSION_ERROR in response to the server headers.
According to the Via header, in all of those cases, the client is an intermediary, 'Websense Web Security Gateway v7.6' (looks like a transparent caching solution)
The RFC says that COMPRESSION_ERROR can be sent in 2 cases.
1) A decoding error in a header block MUST be treated as a connection error of type COMPRESSION_ERROR
2) An endpoint receiving HEADERS, PUSH_PROMISE, or CONTINUATION frames needs to reassemble header blocks and perform decompression even if
the frames are to be discarded. A receiver MUST terminate the connection with a connection error of type COMPRESSION_ERROR if it does not decompress a header block
So in my case it's either an error on my side (bad headers encoding) or it's not an error but the client just discarded the response and sent a COMPRESSION_ERROR as specified by RFC.
If it's the latter then I don't need to fix anything.
But if it's the former then it's a bug in our implementation.
I'd like to rule out the bug case.
Below is a sample of such a session terminated with a COMPRESSION_ERROR.
It's hex dumps of data with my interpretation in {}.
S>C stands for server->client direction, C>S - client->server direction.
Can someone review the dump and tell me if it has some obvious issues with S>C frames or my interpretation of the data?
{preface stripped}
========== S>C frame: ============
00000000: 000024040000000000 ..$......
{type=SETTINGS stream=0 flags=NO_FLAGS(0) len=36 data={...}}
00000000: 00020000000000010000100000030000006400047fffffff000600007f380005 .................d...........8..
00000020: 00007ff7 ....
==================================
========== C>S frame: ============
00000000: 000012040000000000 .........
{type=SETTINGS stream=0 flags=NO_FLAGS(0) len=18 data={...}}
00000000: 000200000000000300000064000600008000 ...........d......
==================================
========== S>C frame: ============
00000000: 000000040100000000 .........
{type=SETTINGS stream=0 flags=ACK(1) len=0 data={}}
==================================
========== C>S frame: ============
00000000: 000004080000000000 .........
{type=WINDOW_UPDATE stream=0 flags=NO_FLAGS(0) len=4 data={inc=10485760}}
00000000: 00a00000 ....
==================================
========== C>S frame: ============
00000000: 000000040100000000 .........
{type=SETTINGS stream=0 flags=ACK(1) len=0 data={}}
==================================
========== C>S frame: ============
00000000: 00015c01050000000d ..\......
{type=HEADERS stream=13 flags=END_STREAM|END_HEADERS(5) len=348 data={...}}
00000000: 418aa4b2186cae824952fdea82449360d4cccb02005f603109a637e36a7c475e A....l..IR...D.`....._`1..7.j|G^
00000020: baa687539e352398ac782c75fd1a91cc56075d537d1a91cc563e7ebe58f9fbed ...S.5#..x,u....V.]S}...V>~.X...
00000040: 00177b73dc9d29ad171863c78f0ba0b54875fbd4c7f96a698a92c86b698072ca ..{s..)...c.....Hu....ji...ki.r.
00000060: 6a0c3497e2d4d3120f6d842c03210a8426c1c85f81cb2c4834903cefede170ed j.4......m.,.!..&.._..,H4.<...p.
00000080: 665435fb875c9b1a7b2dfceda35f7a67967e7ea3a19fde8dd02fdbf97d5e2ee9 fT5..\..{-..._zg.~~....../..}^..
000000a0: c97ad5d07f66a281b0dae053fae46aa43f8429a77a8102e0fb5396ae4e35fda9 .z...f.....S..j.?.).z....S..N5..
000000c0: 0d75d05e458f3192c36cbabb2e29fd66c7bf467fa5283752a988a4ea7fed4bd3 .u.^E.1..l...).f..F..(7R......K.
000000e0: d87a4ac3acae05d971e65708195370e51d8661b65d5d973f508d9bd9abfa5242 .zJ.....q.W..Sp...a.]].?P.....RB
00000100: cb40d25fa523b351b82d4b70ddf45abefb4005dffaee6fbed00177bebdcff7da .#._.#.Q.-Kp..Z..#....o...w.....
00000120: 002eefd7b9d6bda7efb4005dcfaf73ad8f37df6800bb7f5ee75b7f2fbed00176 ...........]..s..7.h...^.[./...v
00000140: bf7c99e458d05a9055391635371496d864fa5310d25f03f4a7775773 .|..X.Z.U9.57...d.S.._...wWs
==================================
========== S>C frame: ============
00000000: 00007c01040000000d ..|......
{type=HEADERS stream=13 flags=END_HEADERS(4) len=124 data={...}}
00000000: 3fe11f887684aa6355e76196c361be940bea435d8a08017d4006e32edc684a62 ?...v..cU.a..a....C]...}#....hJb
00000020: d1bf5f87352398ac5754df0f0d83644f3f6c96df3dbf4a09a5328ea50400bea0 .._.5#..WT....dO?l..=.J..2......
00000040: 1ab826ee32da98b46f628cfe5b234fb2566558923ffcff0f1596dd6d5f4a09e5 ..&.2...ob..[#O.VeX.?......m_J..
00000060: 21aec50400bea05ab8cbf700253168df588aa47e561cc581c034f001 !......Z....%1h.X..~V....4..
==================================
========== S>C frame: ============
00000000: 000cd900010000000d .........
{type=DATA stream=13 flags=END_STREAM(1) len=3289 data={...}}
{data stripped}
==================================
========== C>S frame: ============
00000000: 000014070000000000 .........
{type=GOAWAY stream=0 flags=NO_FLAGS(0) len=20 data={...}}
00000000: 0000000000000009000000000000033200000333 ...............2...3
{last_stream_id=0, error=COMPRESSION_ERROR}
==================================
========== S>C frame: ============
00000000: 000008070000000000 .........
{type=GOAWAY stream=0 flags=NO_FLAGS(0) len=8 data={...}}
00000000: 0000000d00000000 ........
==================================
C>S headers decoded:
:path: /img3/2019/1_24/TH4hGo.png HTTP/2.0
:authority: media3.scdn.vn
Accept: image/webp,image/apng,image/*,*/*;q=0.8
Referer: https://www.sendo.vn/?utm_medium=affiliate&utm_source=accesstrade&aff_sid=...
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,vi;q=0.8,zh;q=0.7,zh-CN;q=0.6,zh-HK;q=0.5,zh-TW;q=0.4
Via: Websense Web Security Gateway v7.6
S>C HEADERS frame decoded:
3fe11f
{set dyn table size 4096}
{dyn table size=0/4096}
88
{indexed header ix=8 :status: 200}
{dyn table not changed}
7684aa6355e7
{literal indexed header ix=54 Server: nginx }
{dyn table size=43/4096:
ix=62 off=0 size=43 Server: nginx
}
6196c361be940bea435d8a08017d4006e32edc684a62d1bf
{literal indexed header ix=33 Date: Fri, 19 Apr 2019 01:37:42 GMT }
{dyn table size=108/4096:
ix=62 off=0 size=65 Date: Fri, 19 Apr 2019 01:37:42 GMT
ix=63 off=65 size=43 Server: nginx
}
5f87352398ac5754df
{literal indexed header ix=31 Content-Type: image/png}
{dyn table size=161/4096:
ix=62 off=0 size=53 Content-Type: image/png
ix=63 off=53 size=65 Date: Fri, 19 Apr 2019 01:37:42 GMT
ix=64 off=118 size=43 Server: nginx
}
0f0d83644f3f
{literal not indexed header ix=28 Content-Length: 3289}
{dyn table not changed}
6c96df3dbf4a09a5328ea50400bea01ab826ee32da98b46f
{literal indexed header ix=44 Last-Modified: Thu, 24 Jan 2019 04:25:35 GMT }
{dyn table size=235/4096:
ix=62 off=0 size=74 Last-Modified: Thu, 24 Jan 2019 04:25:35 GMT
ix=63 off=74 size=53 Content-Type: image/png
ix=64 off=127 size=65 Date: Fri, 19 Apr 2019 01:37:42 GMT
ix=65 off=192 size=43 Server: nginx
}
628cfe5b234fb2566558923ffcff
{literal indexed header ix=34 ETag: "5c493e3f-cd9"}
{dyn table size=285/4096:
ix=62 off=0 size=50 ETag: "5c493e3f-cd9"
ix=63 off=50 size=74 Last-Modified: Thu, 24 Jan 2019 04:25:35 GMT
ix=64 off=124 size=53 Content-Type: image/png
ix=65 off=177 size=65 Date: Fri, 19 Apr 2019 01:37:42 GMT
ix=66 off=242 size=43 Server: nginx
}
0f1596dd6d5f4a09e521aec50400bea05ab8cbf700253168df
{literal not indexed header ix=36 Expires: Sun, 28 Apr 2019 14:39:02 GMT}
{dyn table not changed}
588aa47e561cc581c034f001
{literal indexed header ix=24 Cache-Control: max-age=604800}
{dyn table size=344/4096:
ix=62 off=0 size=59 Cache-Control: max-age=604800
ix=63 off=59 size=50 ETag: "5c493e3f-cd9"
ix=64 off=109 size=74 Last-Modified: Thu, 24 Jan 2019 04:25:35 GMT
ix=65 off=183 size=53 Content-Type: image/png
ix=66 off=236 size=65 Date: Fri, 19 Apr 2019 01:37:42 GMT
ix=67 off=301 size=43 Server: nginx
}
The decoded S>C headers:
:status: 200
Server: nginx
Date: Fri, 19 Apr 2019 01:37:42 GMT
Content-Type: image/png
Content-Length: 3289
Last-Modified: Thu, 24 Jan 2019 04:25:35 GMT
ETag: "5c493e3f-cd9"
Expires: Sun, 28 Apr 2019 14:39:02 GMT
Cache-Control: max-age=604800
I added an hpack decoder to h2get here: https://github.com/deweerdt/h2get/pull/18/files and i'm seeing the same results as you are:
$ ./hpack_decoder 3fe11f887684aa6355e76196c361be940bea435d8a08017d4006e32edc684a62d1bf5f87352398ac5754df0f0d83644f3f6c96df3dbf4a09a5328ea50400bea01ab826ee32da98b46f628cfe5b234fb2566558923ffcff0f1596dd6d5f4a09e521aec50400bea05ab8cbf700253168df588aa47e561cc581c034f001
dyn table resize to 4096
[:status] => [200]
[server] => [nginx]
[date] => [Fri, 19 Apr 2019 01:37:42 GMT]
[content-type] => [image/png]
[content-length] => [3289]
[last-modified] => [Thu, 24 Jan 2019 04:25:35 GMT]
[etag] => ["5c493e3f-cd9"]
[expires] => [Sun, 28 Apr 2019 14:39:02 GMT]
[cache-control] => [max-age=604800]
In Firefox 64 on macOS, when I view one of the disk cache entries under about:cache I see a page that looks something like the code I pasted below. I have deleted most of the long string of code after "security-info" and all but the first three lines of the hex and ascii codes at the end.
How can I restore the content of the file into a human readable format?
I have found quite a few solutions which require tools that only run under Windows, but I'm on macOS. I have also tried to save the hex code (e.g. for the first line: 00000000: 1f 8b 08 00 00 00 00 00 00 03 ed bd fb 72 db 48) into a file with UTF-8 encoding and UNIX linefeeds and .gzip as the file ending and then unpacking it, but all I got was the same hex code again. Do I need to delete the 00000000: at the beginning? Use another file format? Another file ending and/or decoder?
Cache entry information
key: https://www.some-site.com/
fetch count: 2
last fetched: 2018-12-16 20:17:21
last modified: 2018-12-17 06:31:53
expires: Expired Immediately
Data size: 17911 B
Security: This is a secure document.
strongly-framed: 1
security-info: FnhllAKWRHGAlo+ESXykKAAAAAAAAAAAwAAAAAAAAEap ... (and so on)
request-method: POST
request-Accept-Encoding: gzip, deflate, br
request-User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:64.0) Gecko/20100101 Firefox/64.0
response-head: HTTP/1.1 200 OK
Server: Server
Date: Sun, 16 Dec 2018 19:17:14 GMT
Content-Type: text/html; charset=utf-8
Status: 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Request-Id: 2K84AFOOBARX4FNG9D4
ETag: W/"76e8cacfoobar0b8be5"
Cache-Control: max-age=0, private, must-revalidate
X-Runtime: 0.300085
X-Content-Type-Options: nosniff, nosniff
Content-Encoding: gzip
x-amz-rid: 2K84AFOOBARX4FNG9D4
Vary: Accept-Encoding,User-Agent
original-response-headers: Server: Server
Date: Sun, 16 Dec 2018 19:17:14 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Status: 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Request-Id: 2K84AFOOBARX4FNG9D4
ETag: W/"76e8cacfoobar0b8be5"
Cache-Control: max-age=0, private, must-revalidate
X-Runtime: 0.300085
X-Content-Type-Options: nosniff
Content-Encoding: gzip
Set-Cookie: _session_id2=9f298foobarfb208b; path=/; expires=Mon, 17 Dec 2018 01:17:14 -0000; HttpOnly
x-amz-rid: 2K84AFOOBARX4FNG9D4
Vary: Accept-Encoding,User-Agent
net-response-time-onstart: 1266
net-response-time-onstop: 1274
00000000: 1f 8b 08 00 00 00 00 00 00 03 ed bd fb 72 db 48 .............r.H
00000010: 92 2f fc f7 f8 29 d0 9c d9 96 fc b5 78 27 75 b3 ./...)......x'u.
00000020: 25 1f 59 b6 dc ee 76 bb dd 96 3c 9e 6e af 83 01 %.Y...v...<.n...
... (and so on)
You need to extract the hexadecimal from between the colon and the dots or whatever on the right, and convert that back to binary. You will then have, in the case shown, a gzip stream that can be decompressed by gzip. You can use xxd -r -p to convert the hex into binary.
I create one proxy cache using (tor + privoxy + squid) follow
How to create an advanced proxy server using Tor, Privoxy, and Squid
I try call to Proxy cache
curl -x 'my-ip:3400' -I 'https://github.global.ssl.fastly.net/images/modules/dashboard/bootcamp/octocat_setup.png'
curl -x 'my-ip:3400' -I http://hadoop.apache.org/images/hadoop-logo.jpg
https request is ok (through Pivoxy)
HTTP/1.1 200 Connection established
Proxy-Agent: Privoxy/3.0.21
HTTP/1.1 200 OK
Date: Sat, 08 Mar 2014 04:38:37 GMT
Server: GitHub.com
Content-Type: image/png
Last-Modified: Fri, 16 Aug 2013 09:42:18 GMT
Expires: Sun, 08 Mar 2015 04:38:37 GMT
Cache-Control: max-age=31536000, public
Timing-Allow-Origin: https://github.com
Content-Length: 10722
Accept-Ranges: bytes
Via: 1.1 varnish
Age: 0
X-Served-By: cache-am72-AMS
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S1394253517.449177980,VS0,VE97
Vary: Accept-Encoding
but http requets not working
HTTP/1.0 200 OK
Date: Sat, 08 Mar 2014 04:39:19 GMT
Server: Apache/2.4.8-dev (Unix) mod_wsgi/3.4 Python/2.7.5 OpenSSL/1.0.1e
Last-Modified: Tue, 12 Feb 2008 17:38:57 GMT
ETag: "24e3-445f987fe69d8"
Accept-Ranges: bytes
Content-Length: 9443
Content-Type: image/jpeg
X-Cache: MISS from SVR226L-696.localdomain
X-Cache-Lookup: MISS from SVR226L-696.localdomain:3128
Via: 1.0 SVR226L-696.localdomain (squid)
Connection: keep-alive
Please suggest a specific solution
with Open::URI, I can do the following:
require 'open-uri'
#check status
open('http://google.com').status
#get entire html
open('http://google.com').read
Is it possible to get the HTTP headers of a request so things can be debugged, something like Curls' curl -I http://google.com?
$ curl -I google.com
HTTP/1.1 301 Moved Permanently
Location: http://www.google.com/
Content-Type: text/html; charset=UTF-8
Date: Mon, 17 Dec 2012 14:28:17 GMT
Expires: Wed, 16 Jan 2013 14:28:17 GMT
Cache-Control: public, max-age=2592000
Server: gws
Content-Length: 219
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Is this possible?
Use the meta method of the virtual filehandle:
open('http://google.com'){|f| pp f.meta }
{"x-frame-options"=>"SAMEORIGIN",
"expires"=>"-1",
"p3p"=>
"CP=\"This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info.\"",
"content-type"=>"text/html; charset=ISO-8859-1",
"date"=>"Mon, 17 Dec 2012 14:37:29 GMT",
"server"=>"gws",
"x-xss-protection"=>"1; mode=block",
"set-cookie"=>
"PREF=ID=d2fb8a93d369bcd2:FF=0:TM=1355755049:LM=1355755049:S=ONVSP6n2jtluFgll; expires=Wed, 17-Dec-2014 14:37:29 GMT; path=/; domain=.google.com, NID=67=OFEvvHCOa3C6wScQCUIKfu_89oL9MSmnFjwN-u5LX_foP8NLsX7G9dq48NLVrf4WUXhqOA1jb38s0e9qeRp1Iwx_LT_N8IuF0Qi6dXVtR2zdvA86INqtfg5uNrKvxJfJ; expires=Tue, 18-Jun-2013 14:37:29 GMT; path=/; domain=.google.com; HttpOnly",
"cache-control"=>"private, max-age=0",
"transfer-encoding"=>"chunked"}
http://www.ruby-doc.org/stdlib-1.9.3/libdoc/open-uri/rdoc/OpenURI/Meta.html