I am trying to understand the way the HTTP/2 protocol compresses the Data field. I didn't find an algorithm in the rfc. I know its a binary protocol.
I am looking for a way to get the binary protocol back into human readable content. I thought it is gzip like writen in the header but it isn't anyone has a source where i can look for the binary protocol refernce?
Frame 55: 151 bytes on wire (1208 bits), 151 bytes captured (1208 bits) on interface 0
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst:00:00:00_00:00:00 (00:00:00:00:00:00)
Internet Protocol Version 4, Src: 127.0.0.1, Dst: 127.0.0.1
Transmission Control Protocol, Src Port: 443 (443), Dst Port: 55300 (55300), Seq: 1087, Ack: 1078, Len: 85
Secure Sockets Layer
HyperText Transfer Protocol 2
Stream: HEADERS, Stream ID: 13, Length 47
Length: 47
Type: HEADERS (1)
Flags: 0x04
0... .... .... .... .... .... .... .... = Reserved: 0x00000000
.000 0000 0000 0000 0000 0000 0000 1101 = Stream Identifier: 13
[Pad Length: 0]
Header Block Fragment: 8854012a5a839bd9ab5f87497ca589d34d1f5c0333333861...
[Header Length: 177]
[Header Count: 6]
Header: :status: 200
Header: access-control-allow-origin: *
Header: content-encoding: gzip
Header: content-type: text/html
Header: content-length: 338
Header: date: Wed, 17 Aug 2016 15:14:25 GMT
Padding: <MISSING>
Frame 56: 442 bytes on wire (3536 bits), 442 bytes captured (3536 bits) on interface 0
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
Internet Protocol Version 4, Src: 127.0.0.1, Dst: 127.0.0.1
Transmission Control Protocol, Src Port: 443 (443), Dst Port: 55300 (55300), Seq: 1172, Ack: 1078, Len: 376
Secure Sockets Layer
HyperText Transfer Protocol 2
Stream: DATA, Stream ID: 13, Length 338
Length: 338
Type: DATA (0)
Flags: 0x01
0... .... .... .... .... .... .... .... = Reserved: 0x00000000
.000 0000 0000 0000 0000 0000 0000 1101 = Stream Identifier: 13
[Pad Length: 0]
Data: 1f8b080000096e8800ff9cd2416b1b311005e0b3ffc5eb9e...
Padding: <MISSING>
The response body of a HTTP/2 response is compressed (or not) exactly the same way as it works with HTTP/1: Accept-Encoding: in the request and Content-Encoding: in the response. HTTP/2 did nothing new with compression of the bodies, only with headers which weren't compressed at all in HTTP/1.
Related
I need to reproduce a bug with OkHttp so I can file a bug or ask a question on StackOverflow.
What is the simplest way to do this without a lot of setup?
I've read https://stackoverflow.com/help/how-to-ask and https://stackoverflow.com/help/minimal-reproducible-example but I'm still stuck? Help me!
Make a Kotlin script in Intellij, place it outside any source folders and make sure it ends with .main.kts filename.
example.main.kts
#!/usr/bin/env kotlin
#file:Repository("https://repo1.maven.org/maven2/")
#file:DependsOn("com.squareup.okhttp3:okhttp:4.9.0")
#file:CompilerOptions("-jvm-target", "1.8")
import okhttp3.OkHttpClient
import okhttp3.Request
val client = OkHttpClient()
val request = Request.Builder()
.url("https://raw.github.com/square/okhttp/master/README.md")
.build()
val body = client.newCall(request).execute().use {
it.body!!.string()
}
println(body)
The #! line means it will run like a shell script also
$ ./example.main.kts
OkHttp
======
See the [project website][okhttp] for documentation and APIs.
...
If you need to see events (connection, request, response, caching) then use the EventListener.
https://square.github.io/okhttp/events/
In Gradle add a dependency on com.squareup.okhttp3:logging-interceptor:4.9.1
val client = OkHttpClient.Builder()
.eventListenerFactory(LoggingEventListener.Factory())
.build()
[0 ms] callStart: Request{method=GET, url=https://httpbin.org/get}
[11 ms] proxySelectStart: https://httpbin.org/
[12 ms] proxySelectEnd: [DIRECT]
[12 ms] dnsStart: httpbin.org
[55 ms] dnsEnd: [httpbin.org/54.147.165.197, httpbin.org/34.231.30.52, httpbin.org/54.91.118.50, httpbin.org/18.214.80.1, httpbin.org/54.166.163.67, httpbin.org/34.199.75.4]
[62 ms] connectStart: httpbin.org/54.147.165.197:443 DIRECT
[176 ms] secureConnectStart
[747 ms] secureConnectEnd: Handshake{tlsVersion=TLS_1_2 cipherSuite=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 peerCertificates=[CN=httpbin.org, CN=Amazon, OU=Server CA 1B, O=Amazon, C=US, CN=Amazon Root CA 1, O=Amazon, C=US] localCertificates=[]}
[765 ms] connectEnd: h2
[767 ms] connectionAcquired: Connection{httpbin.org:443, proxy=DIRECT hostAddress=httpbin.org/54.147.165.197:443 cipherSuite=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 protocol=h2}
[775 ms] requestHeadersStart
[783 ms] requestHeadersEnd
[993 ms] responseHeadersStart
[994 ms] responseHeadersEnd: Response{protocol=h2, code=200, message=, url=https://httpbin.org/get}
[999 ms] responseBodyStart
[999 ms] responseBodyEnd: byteCount=267
[999 ms] connectionReleased
[1000 ms] callEnd
[0 ms] callStart: Request{method=GET, url=https://httpbin.org/get}
[1 ms] connectionAcquired: Connection{httpbin.org:443, proxy=DIRECT hostAddress=httpbin.org/54.147.165.197:443 cipherSuite=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 protocol=h2}
[1 ms] requestHeadersStart
[1 ms] requestHeadersEnd
[98 ms] responseHeadersStart
[99 ms] responseHeadersEnd: Response{protocol=h2, code=200, message=, url=https://httpbin.org/get}
[99 ms] responseBodyStart
[99 ms] responseBodyEnd: byteCount=267
[99 ms] connectionReleased
[99 ms] callEnd
If you need to see HTTP/2 Frames https://square.github.io/okhttp/debug_logging/
[2021-02-27 11:49:39] >> CONNECTION 505249202a20485454502f322e300d0a0d0a534d0d0a0d0a
[2021-02-27 11:49:39] >> 0x00000000 6 SETTINGS
[2021-02-27 11:49:39] >> 0x00000000 4 WINDOW_UPDATE
[2021-02-27 11:49:39] << 0x00000000 18 SETTINGS
[2021-02-27 11:49:39] << 0x00000000 4 WINDOW_UPDATE
[2021-02-27 11:49:39] >> 0x00000003 33 HEADERS END_STREAM|END_HEADERS
[2021-02-27 11:49:39] >> 0x00000000 0 SETTINGS ACK
[2021-02-27 11:49:39] << 0x00000000 0 SETTINGS ACK
[2021-02-27 11:49:39] << 0x00000003 113 HEADERS END_HEADERS
[2021-02-27 11:49:39] << 0x00000003 267 DATA
[2021-02-27 11:49:39] << 0x00000003 0 DATA END_STREAM
Follow up request
[2021-02-27 11:49:39] >> 0x00000005 10 HEADERS END_STREAM|END_HEADERS
[2021-02-27 11:49:39] << 0x00000005 113 HEADERS END_HEADERS
[2021-02-27 11:49:39] << 0x00000005 267 DATA
[2021-02-27 11:49:39] << 0x00000005 0 DATA END_STREAM
To create an Android instrumentation test run on device or the emulator, use a test like.
https://github.com/square/okhttp/blob/master/regression-test/src/androidTest/java/okhttp/regression/compare/OkHttpClientTest.java
#RunWith(AndroidJUnit4.class)
public class OkHttpClientTest {
#Test public void get() throws IOException {
OkHttpClient client = new OkHttpClient();
Request request = new Request.Builder()
.url("https://google.com/robots.txt")
.build();
try (Response response = client.newCall(request).execute()) {
assertEquals(200, response.code());
assertEquals(Protocol.HTTP_2, response.protocol());
}
}
}
Steps to run https://github.com/square/okhttp/tree/master/regression-test
If you need to see network traffic via Wireshark for a JVM client (JSSE and TLSv1.2) https://github.com/square/okhttp/blob/master/samples/guide/src/main/java/okhttp3/recipes/kt/WiresharkExample.kt
Copy WireSharkKeyLoggerListener to your test code to use in development.
This logs TLSv1.2 on a JVM (OpenJDK 11+) without any additional code. For TLSv1.3 an existing external tool is required.
Steps to run in your own code
In your main method WireSharkListenerFactory.register()
Create Listener factory val eventListenerFactory = WireSharkListenerFactory( logFile = File("/tmp/key.log"), tlsVersions = tlsVersions, launch = launch)
Register with client.eventListenerFactory(eventListenerFactory)
Launch wireshark if not done externally val process = eventListenerFactory.launchWireShark()
Capturing on 'Wi-Fi: en0'
Frame 20: 110 bytes on wire (880 bits), 110 bytes captured (880 bits) on interface en0, id 0
Ethernet II, Src: Google_fc:86:a2 (28:bd:89:fc:86:a2), Dst: Apple_6b:3b:e5 (f0:18:98:6b:3b:e5)
Internet Protocol Version 4, Src: 104.244.42.130, Dst: 192.168.86.23
Transmission Control Protocol, Src Port: 443, Dst Port: 57669, Seq: 3531, Ack: 383, Len: 44
Transport Layer Security
TLSv1.2 Record Layer: Application Data Protocol: http2
Content Type: Application Data (23)
Version: TLS 1.2 (0x0303)
Length: 39
Encrypted Application Data: cfb509e24e2ca451923820e45c3943fa521e1f3f1a821fe3468c0a7294e1d07c0ab7ab90…
[Application Data Protocol: http2]
HyperText Transfer Protocol 2
Stream: SETTINGS, Stream ID: 0, Length 6
Length: 6
Type: SETTINGS (4)
Flags: 0x00
0000 000. = Unused: 0x00
.... ...0 = ACK: False
0... .... .... .... .... .... .... .... = Reserved: 0x0
.000 0000 0000 0000 0000 0000 0000 0000 = Stream Identifier: 0
Settings - Initial Windows size : 65536
Settings Identifier: Initial Windows size (4)
Initial Windows Size: 65536
...
HyperText Transfer Protocol 2
Stream: HEADERS, Stream ID: 3, Length 53, GET /robots.txt?s=tw
Length: 53
Type: HEADERS (1)
Flags: 0x05, End Headers, End Stream
00.0 ..0. = Unused: 0x00
..0. .... = Priority: False
.... 0... = Padded: False
.... .1.. = End Headers: True
.... ...1 = End Stream: True
0... .... .... .... .... .... .... .... = Reserved: 0x0
.000 0000 0000 0000 0000 0000 0000 0011 = Stream Identifier: 3
[Pad Length: 0]
Header Block Fragment: 82048c62c3c674a174f94ff88813e3418b1d665d3e0c9496c5c87a7f8750839bd9ab7a91…
[Header Length: 166]
[Header Count: 6]
Header: :method: GET
Name Length: 7
...
You will need to point Wireshark at your key log file
Use a MITM debugging proxy, these give you user friendly tools to observe traffic.
https://medium.com/#ievgeniitkachenko/different-ways-to-debug-https-requests-in-an-android-emulator-cbf4e0c8ce17
https://hackupstate.medium.com/using-charles-proxy-to-debug-android-ssl-traffic-e61fc38760f7
When using --negotiate (or ntlm) with curl on windows, SSL/TSL handshake fails despite having a valid kerberos ticket cached on my windows 10 (shown below). The same logic and commands works without any issue in Unix/Linux. Any idea/help on how to resolve this issue?
Klist details:
$: Klist
Client: username # XXXX.XXX
Server: cifs/XXXXXXX.XXX # XXXXXXX.XXX
KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
Ticket Flags XXXXXX -> forwardable renewable pre_authent name_canonicalize
Start Time: 8/27/2020 9:46:36 (local)
End Time: 8/27/2020 19:46:33 (local)
Renew Time: 9/27/2020 9:46:33 (local)
Session Key Type: AES-256-CTS-HMAC-SHA1-96
Cache Flags: 0
Kdc Called: XXXXXXXXX.XXX
curl command using ntlm or negotiate details:
>curl --ntlm -u : https://XXXXX.XXX -v
OR
>curl --negotiate -u : https://XXXXX.XXX -v
* Trying xx.xx.xx.xx...
* TCP_NODELAY set
* Connected to xxxx.xxx (xx.xx.xx.xx) port xxx (#0)
* schannel: SSL/TLS connection with xxxx.xxx port xxx (step 1/3)
* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 186 bytes...
* schannel: sent initial handshake data: sent 186 bytes
* schannel: SSL/TLS connection with xxx.xxx port xxx (step 2/3)
* schannel: failed to receive handshake, need more data
* schannel: SSL/TLS connection with xxx.xxx port xxx (step 2/3)
* schannel: encrypted data got 4096
* schannel: encrypted data buffer: offset 4096 length 4096
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with xxx.xxx port xxx (step 2/3)
* schannel: encrypted data got 1024
* schannel: encrypted data buffer: offset 5120 length 5120
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with xxx.xxx port xxx (step 2/3)
* schannel: encrypted data got 817
* schannel: encrypted data buffer: offset 5937 length 6144
* schannel: sending next handshake data: sending 126 bytes...
* schannel: SSL/TLS connection with xxx.xxx port xxx (step 2/3)
* schannel: encrypted data got 51
* schannel: encrypted data buffer: offset 51 length 6144
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with xxx.xxx port xxx (step 3/3)
* schannel: stored credential handle in session cache
> GET /login HTTP/1.1
> Host: xxx.xxx
> User-Agent: curl/7.55.1
> Accept: */*
>
* schannel: client wants to read 102400 bytes
* schannel: encdata_buffer resized 103424
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: encrypted data got 915
* schannel: encrypted data buffer: offset 915 length 103424
* schannel: decrypted data length: 852
* schannel: decrypted data added: 852
* schannel: decrypted data cached: offset 852 length 102400
* schannel: encrypted data length: 34
* schannel: encrypted data cached: offset 34 length 103424
* schannel: decrypted data length: 5
* schannel: decrypted data added: 5
* schannel: decrypted data cached: offset 857 length 102400
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: decrypted data buffer: offset 857 length 102400
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 857
* schannel: decrypted data buffer: offset 0 length 102400
< HTTP/1.1 401
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< X-Application-Context: Apixxxxx:x
< X-RateLimit-Limit-Api_login_anonymous: 1000
< X-RateLimit-Remaining-Api_login_anonymous: 999
< X-RateLimit-Reset-Apixxx_login_anonymous: 0
< X-xxxxx-xxx: xxxxx.xxx
< Date: Thu, 27 Aug 2020 19:50:12 GMT
< Expires: 0
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: DENY
< WWW-Authenticate: Negotiate
< WWW-Authenticate: Basic realm="Please login with your Windows account"
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Strict-Transport-Security: max-age=xxx ; includeSubDomains
< TraceId: 5f480e7427f78fd5406fcbef0abf8e6c
< X-Content-Type-Options: nosniff
< Transfer-Encoding: chunked
<
* Ignoring the response-body
* Connection #0 to host xxx.xxx left intact
* Issue another request to this URL: 'https://xxx.xxx/login'
* Found bundle for host xxx.xxx: 0xxxxxxxxx [can pipeline]
* Re-using existing connection! (#0) with host xxx.xxx
* Connected to xxx.xxx (xx.xx.xxx.xx) port xxx (#0)
* Server auth using Negotiate with user ' '
> GET /login HTTP/1.1
> Host: xxx.xxx
> Authorization: Negotiate xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==
> User-Agent: curl/7.55.1
> Accept: */*
>
* schannel: client wants to read 102400 bytes
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: encrypted data got 814
* schannel: encrypted data buffer: offset 814 length 103424
* schannel: decrypted data length: 751
* schannel: decrypted data added: 751
* schannel: decrypted data cached: offset 751 length 102400
* schannel: encrypted data length: 34
* schannel: encrypted data cached: offset 34 length 103424
* schannel: decrypted data length: 5
* schannel: decrypted data added: 5
* schannel: decrypted data cached: offset 756 length 102400
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: decrypted data buffer: offset 756 length 102400
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 756
* schannel: decrypted data buffer: offset 0 length 102400
< HTTP/1.1 401
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< X-Application-Context: Apixxx:xxxxxxxxxxxxxxxx:x
< X-RateLimit-Limit-Apixxx_login_anonymous: 1000
< X-RateLimit-Remaining-Apixxx_login_anonymous: 999
< X-RateLimit-Reset-Apixxx_login_anonymous: 0
< X-XXXX-xxx: xxxx.xxx
< Date: Thu, 27 Aug 2020 19:50:12 GMT
< Expires: 0
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: DENY
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Strict-Transport-Security: max-age=xxx ; includeSubDomains
< TraceId: xxxxxxxxxxxxxxxxxxxxxxxxxxx
< X-Content-Type-Options: nosniff
< Transfer-Encoding: chunked
<
* Connection #0 to host xxx.xxx left intact
>curl --version
curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
Release-Date: [unreleased]
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL
I've been using cURL for a while now, though not upgrading the executable very often. I've noticed with v7.60 that when using -v to get the headers as well, I get a smattering of * schannel: messages in the prompt, so that it goes like this:
$ curl -vs "https://example.com"
* Rebuilt URL to: https://example.com/
* Trying 93.184.216.34...
* TCP_NODELAY set
* Connected to example.com (93.184.216.34) port 443 (#0)
* schannel: SSL/TLS connection with example.com port 443 (step 1/3)
* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 176 bytes...
* schannel: sent initial handshake data: sent 176 bytes
* schannel: SSL/TLS connection with example.com port 443 (step 2/3)
* schannel: failed to receive handshake, need more data
* schannel: SSL/TLS connection with example.com port 443 (step 2/3)
* schannel: encrypted data got 4096
* schannel: encrypted data buffer: offset 4096 length 4096
* schannel: encrypted data length: 14
* schannel: encrypted data buffer: offset 14 length 4096
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with example.com port 443 (step 2/3)
* schannel: encrypted data got 817
* schannel: encrypted data buffer: offset 831 length 4096
* schannel: sending next handshake data: sending 126 bytes...
* schannel: SSL/TLS connection with example.com port 443 (step 2/3)
* schannel: encrypted data got 242
* schannel: encrypted data buffer: offset 242 length 4096
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with example.com port 443 (step 3/3)
* schannel: stored credential handle in session cache
> GET / HTTP/1.1
> Host: example.com
> User-Agent: curl/7.60.0
> Accept: */*
>
* schannel: client wants to read 102400 bytes
* schannel: encdata_buffer resized 103424
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: encrypted data got 1666
* schannel: encrypted data buffer: offset 1666 length 103424
* schannel: decrypted data length: 338
* schannel: decrypted data added: 338
* schannel: decrypted data cached: offset 338 length 102400
* schannel: encrypted data length: 1299
* schannel: encrypted data cached: offset 1299 length 103424
* schannel: decrypted data length: 1270
* schannel: decrypted data added: 1270
* schannel: decrypted data cached: offset 1608 length 102400
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: decrypted data buffer: offset 1608 length 102400
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 1608
* schannel: decrypted data buffer: offset 0 length 102400
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Cache-Control: max-age=604800
< Content-Type: text/html; charset=UTF-8
< Date: Tue, 05 Feb 2019 08:08:57 GMT
< Etag: "1541025663"
< Expires: Tue, 12 Feb 2019 08:08:57 GMT
< Last-Modified: Fri, 09 Aug 2013 23:54:35 GMT
< Server: ECS (dca/24C1)
< Vary: Accept-Encoding
< X-Cache: HIT
< Content-Length: 1270
<
[HTML here]
Is there any way to suppress these messages so that -v still yields me the headers, and I can use > NUL to discard the response body, all without the SChannel noise?
To head off the obvious suggestion, -I won't work for me: I need to be able to send GET and other requests this way because my work involves investigating third-party servers that can (and have been known to) respond to HEAD differently.
Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers.
This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.
Closed 4 years ago.
Improve this question
I have a proxy that acts between a Minecraft (Windows 10 Edition) client and a server, the protocol is UDP. The client broadcasts an unconnected ping, then server replies with an unconnected pong, everything with that is okay. The thing is when the server sends the packet to my proxy, my proxy sends it to client but for some reason something happens during that part; either my proxy is not sending the packet or the client is not receiving the packet, but most likely it's the second option.
Edit: I got the unconnected pings and pongs working, now the server shows online on the server list, now the problem is mainly the open connection requests/replies. How I got the the pings and pongs working was I re-encoded the buffers and send them instead of sending them raw.
Here you can see from wireshark, the minecraft client sends an unconnected ping the proxy:
NO. Time. Source. Dest. Proto. Len. Packet
417 10.452413 10.0.0.248 10.0.0.255 RakNet 75 Unconnected Ping (client -> proxy)
430 10.457000 10.0.0.248 x.x.x.x RakNet 610 Unconnected Ping (proxy -> server)
431 10.587214 x.x.x.x 10.0.0.248 RakNet 212 Unconnected Pong (server -> proxy -> client)
Now the proxy receive the unconnected pong, send it to the client, and the client doesn't receive it, I can confirm this because on the game the server shows offline and doesn't show any data:
Localhost Server Screenshot
In my code, I first bind the proxy on port 19132 and set the server address I want to communicate with:
var config = NewConfig()
var proxy = Proxy{}
var err error
proxy.UDPConn, err = net.ListenUDP("udp", &net.UDPAddr{IP: net.ParseIP(config.BindAddr), Port: config.BindPort})
if err != nil {
Panic(err.Error())
os.Exit(1)
}
Info("Starting proxy on " + config.BindAddr + ":" + strconv.Itoa(config.BindPort))
addrs, err := net.LookupHost(config.ServerAddr)
if err != nil {
Panic(err.Error())
os.Exit(1)
}
conn := NewConnection(&proxy)
conn.server.SetAddress(net.UDPAddr{IP: net.ParseIP(addrs[0]), Port: config.ServerPort})
conn.HandleIncomingPackets()
Now my proxy starts receiving and sending packets automatically.
for true {
buffer := make([]byte, 2048)
_, addr, err := conn.proxy.UDPConn.ReadFromUDP(buffer)
if err != nil {
Alert(err.Error())
continue
}
MessageId := buffer[0]
Debug("Message Id : " + strconv.Itoa(int(MessageId))) // this is the packet id
if conn.client.IsConnected() { // client is connected
if conn.IsServer(*addr) {
conn.pkHandler.HandleIncomingPacket(buffer, conn.client) // if server send to client
}else{
conn.pkHandler.HandleIncomingPacket(buffer, conn.server) // if client send to server
}
} else {
switch MessageId {
case byte(IdUnconnectedPingOpenConnection):
conn.handleUnconnectedPing(*addr, buffer) // send this server
break
case byte(IdUnconnectedPongOpenConnection):
conn.handleUnconnectedPong(*addr, buffer) // parse server data and send to client
break
case byte(IdOpenConnectionRequest1):
conn.handleConnectionRequest1(*addr, buffer) // connect client and send to server
break
//case byte(IdOpenConnectionReply1):
// conn.handleConnectionReply1(*addr, buffer)
// break
}
}
}
This is the message id log:
[2018-06-10 13:52:12][Log/DEBUG]: Message Id : 1
[2018-06-10 13:52:12][Log/INFO]: Received unconnected ping from client address: 10.0.0.248
[2018-06-10 13:52:12][Log/DEBUG]: Message Id : 28
[2018-06-10 13:52:12][Log/INFO]: Received unconnected pong from server address: x.x.x.x
[2018-06-10 13:52:13][Log/DEBUG]: Message Id : 1
[2018-06-10 13:52:13][Log/INFO]: Received unconnected ping from client address: 10.0.0.248
[2018-06-10 13:52:13][Log/DEBUG]: Message Id : 28
[2018-06-10 13:52:13][Log/INFO]: Received unconnected pong from server address: x.x.x.x
Another way I confirmed the client is not receiving packets is that when in the game I click the server in the server list, the client sends an open connection request 1 (5) and the server replies with open connection reply 1 (6), the client is supposed to receive this and continue with an open connection request 2 (7) then finally the server replies with open connection reply 2 (8), but the client never sends open connection request 2 (7), because it never got the open connection reply 1 (6) from the proxy and hence times out and disconnects, here is a log showing this:
[2018-06-10 11:07:46][Log/DEBUG]: Message Id : 5
[5 0 255 255 0 254 254 254 254 253 253 253 253 18 52 86 120 8]
[2018-06-10 11:07:47][Log/DEBUG]: Message Id : 6
[6 0 255 255 0 254 254 254 254 253 253 253 253 18 52 86 120 36 149 162 237 197 55 226 161 0 8 28]
[2018-06-10 11:07:47][Log/DEBUG]: Message Id : 5
[5 0 255 255 0 254 254 254 254 253 253 253 253 18 52 86 120 8]
[2018-06-10 11:07:47][Log/DEBUG]: Message Id : 6
[6 0 255 255 0 254 254 254 254 253 253 253 253 18 52 86 120 36 149 162 237 197 55 226 161 0 8 28]
[2018-06-10 11:07:47][Log/DEBUG]: Message Id : 1
[1 0 0 0 0 3 3 91 191 0 255 255 0 254 254 254 254 253 253 253 253 18 52 86 120 191 216 14 215 31 123 8 249]
After so much debugging and testing, I figured I just needed to continue sending the datagram buffers, if it came from the server send it to client and vice-versa. If I cancelled the packet from sending I needed to send an ACK. It was all that simple.
I was inspecting the websocket traffic between my server and my browser with Wireshark when I noticed this kind of frame :
No. Time Source Destination Protocol Length Info
144342 8212.033150000 127.0.0.1 127.0.0.1 WebSocket 821 WebSocket Text [FIN]
Frame 144342: 821 bytes on wire (6568 bits), 821 bytes captured (6568 bits) on interface 0
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
Internet Protocol Version 4, Src: 127.0.0.1 (127.0.0.1), Dst: 127.0.0.1 (127.0.0.1)
Transmission Control Protocol, Src Port: http-alt (8080), Dst Port: 53749 (53749), Seq: 1132, Ack: 603, Len: 755
WebSocket
1... .... = Fin: True
.000 .... = Reserved: 0x00
.... 0001 = Opcode: Text (1)
0... .... = Mask: False
.111 1110 = Payload length: 126 Extended Payload Length (16 bits)
Extended Payload length (16 bits): 140
Payload
Text: {"type":"COLLABORATIVE_COMMANDS","remoteUser":"null","key":"1c78c08f-5d2d-445a-a63c-3a211d2f0336","messageBroadcast":{"smartPath":["null"]}}
WebSocket
1... .... = Fin: True
.000 .... = Reserved: 0x00
.... 0001 = Opcode: Text (1)
0... .... = Mask: False
.111 1110 = Payload length: 126 Extended Payload Length (16 bits)
Extended Payload length (16 bits): 329
Payload
Text [truncated]: {"type":"COLLABORATIVE_COMMANDS","remoteUser":"","key":"1c78c08f-5d2d-445a-a63c-3a211d2f0336","messageBroadcast":{"cameraInfos":{"target":{"x":0,"y":0,"z":0},"camPos":{"x":557.0133301398326,"y":159.5460628202445,"z":342.4
WebSocket
1... .... = Fin: True
.000 .... = Reserved: 0x00
.... 0001 = Opcode: Text (1)
0... .... = Mask: False
.111 1110 = Payload length: 126 Extended Payload Length (16 bits)
Extended Payload length (16 bits): 141
Payload
Text: {"type":"COLLABORATIVE_COMMANDS","remoteUser":"","key":"1c78c08f-5d2d-445a-a63c-3a211d2f0336","messageBroadcast":{"colourEditedMeshes":true}}
WebSocket
1... .... = Fin: True
.000 .... = Reserved: 0x00
.... 0001 = Opcode: Text (1)
0... .... = Mask: False
.111 1110 = Payload length: 126 Extended Payload Length (16 bits)
Extended Payload length (16 bits): 129
Payload
Text: {"type":"COLLABORATIVE_COMMANDS","remoteUser":"","key":"1c78c08f-5d2d-445a-a63c-3a211d2f0336","messageBroadcast":{"explode":"0"}}
Does this mean there are several websockets in my packet ? How is this possible ?
If you read the WebSocket spec, RFC 6455, you will see that WebSocket packets are framed, where each frame has its own header and payload. Remember that TCP is a streaming transport. Senders and receivers are not paying attention to the TCP frames, they are paying attention to the payloads within those frames. A WebSocket sender will send a WebSocket header followed by its payload, followed by the next WebSocket header and its payload, and so on. A WebSocket receiver will read a WebSocket header and its payload, then read the next WebSocket header and its payload, and so on. Typically, the Nagle algorithm is enabled on TCP sockets, and it will split and combine application data into TCP frames as needed for efficient network transmissions. That is handled by the TCP stack transparent to the applications. So yes, it is possible to see multiple WebSocket frames appear inside of a single TCP frame, if that is how Nagle decided to transmit them. If the WebSocket packets are sent in a short period of time, Nagle may merge them so it only has to transmit a single TCP frame instead of separate TCP frames.