Websocket client hanshake error 426 on upgrade - websocket

Reader's digest version:
First time writing a websocket client. Java 1.8 using Eclipse. While on my workplace internal network getting a handshake error exception and the websocket log information tells me:
< Session 1c61123c-aa5f-4e59-a49d-b5d3c77c0bbd [598 ms]: Received handshake response:
< 426
< connection: Keep-Alive
< content-length: 29
< content-type: text/plain; charset=utf-8
< date: Mon, 12 Jun 2017 16:40:04 GMT
< server: nginx
Exception in thread "main" java.lang.RuntimeException: javax.websocket.DeploymentException: Handshake error.
at webSocketExample.WebsocketClientEndpoint.<init>(WebsocketClientEndpoint.java:60)
Details:
When I am connected to my internal work network and try to make a websocket connection I initially get connected but then I am sending a TCP reset. If I throw the code on my laptop and connect over a different network like my home Comcast connection the connection is made and I receive the initial data response from the server. I then take my laptop and hook it up to my internal work network and I go back to getting the handshake error.
I have been unable to resolve it yet with my internal security team as we don't do websocket development here so the knowledge base is a little lacking.
I set my proxy prior to calling this routine from my main. It has been verified that I am getting connected to the server but then I am sending a reset. Here is the code I am executing:
try {
System.setProperty("javax.net.debug", "all");
ClientManager client = ClientManager.createClient();
final SSLContextConfigurator defaultConfig = new SSLContextConfigurator();
defaultConfig.retrieve(System.getProperties());
SSLEngineConfigurator sslEngine = new SSLEngineConfigurator(defaultConfig, true, false, false);
client.getProperties().put(ClientProperties.SSL_ENGINE_CONFIGURATOR, sslEngine);
client.getProperties().put(ClientProperties.LOG_HTTP_UPGRADE, true);
client.connectToServer(new Endpoint() {
#Override
public void onOpen(Session session, EndpointConfig config) {
session.addMessageHandler(new MessageHandler.Whole<String>() {
#Override
public void onMessage(String message) {
System.out.println(message);
}
});
}
}, ClientEndpointConfig.Builder.create().build(),
new URI("wss://api.sandbox.gemini.com/v1/marketdata/BTCUSD"));
} catch (Exception e) {
throw new RuntimeException(e);
}
I have read all the posts/questions/answers that I have been able to find but haven't come up with anything yet. I am somewhat limited on what I can personally do on our network as we have pretty tight security but I can work with my security team to find answers if I get asked a question.
I would think it has to be internal to my work network and the proxy maybe but I haven't been able to resolve it. Also, this is the last part of the log information from the connection and error reporting:
Padded plaintext after DECRYPTION: len = 170
0000: 48 54 54 50 2F 31 2E 31 20 34 32 36 20 55 70 67 HTTP/1.1 426 Upg
0010: 72 61 64 65 20 52 65 71 75 69 72 65 64 0D 0A 44 rade Required..D
0020: 61 74 65 3A 20 4D 6F 6E 2C 20 31 32 20 4A 75 6E ate: Mon, 12 Jun
0030: 20 32 30 31 37 20 31 36 3A 35 30 3A 32 35 20 47 2017 16:50:25 G
0040: 4D 54 0D 0A 53 65 72 76 65 72 3A 20 6E 67 69 6E MT..Server: ngin
0050: 78 0D 0A 43 6F 6E 6E 65 63 74 69 6F 6E 3A 20 4B x..Connection: K
0060: 65 65 70 2D 41 6C 69 76 65 0D 0A 43 6F 6E 74 65 eep-Alive..Conte
0070: 6E 74 2D 54 79 70 65 3A 20 74 65 78 74 2F 70 6C nt-Type: text/pl
0080: 61 69 6E 3B 20 63 68 61 72 73 65 74 3D 75 74 66 ain; charset=utf
0090: 2D 38 0D 0A 43 6F 6E 74 65 6E 74 2D 4C 65 6E 67 -8..Content-Leng
00A0: 74 68 3A 20 32 39 0D 0A 0D 0A th: 29....
[Raw read (bb)]: length = 58
0000: 17 03 03 00 35 EC 0F C3 40 0D 18 B4 7B 53 C6 2D ....5...#....S.-
0010: F5 83 29 72 15 51 AC 31 8F 86 52 C9 3C 98 BB F8 ..)r.Q.1..R.<...
0020: 94 4A D7 3A 6E 17 8C 75 9E 58 25 0B AE B5 2F 13 .J.:n..u.X%.../.
0030: 8C 44 29 94 2F 4A 46 18 C9 1A .D)./JF...
Padded plaintext after DECRYPTION: len = 29
0000: 55 70 67 72 61 64 65 20 74 6F 20 57 65 62 53 6F Upgrade to WebSo
0010: 63 6B 65 74 20 72 65 71 75 69 72 65 64 cket required
> Session 6f920826-4b05-47bd-8cad-212038ff9fad [86 ms]: Sending handshake request:
> GET wss://api.sandbox.gemini.com/v1/marketdata/BTCUSD
> Connection: Upgrade
> Host: api.sandbox.gemini.com
> Origin: api.sandbox.gemini.com
> Sec-WebSocket-Key: nkgv7uk/bPSKBiOz5T/Bdg==
> Sec-WebSocket-Version: 13
> Upgrade: websocket
< Session 6f920826-4b05-47bd-8cad-212038ff9fad [600 ms]: Received handshake response:
< 426
< connection: Keep-Alive
< content-length: 29
< content-type: text/plain; charset=utf-8
< date: Mon, 12 Jun 2017 16:50:25 GMT
< server: nginx
If you got this far, thanks...
Adding sniffer information from my network team:
You establish a connection Pkt 263, which you are going through the proxy server, mcweb-a 170.137.249.117. Then you transmit data. The reset is sent in Pkt 280.
enter image description here

In case anyone comes across this. The issue was a problem with my proxy server. We had to add the specific endpoint as being allowable so it would skip some of the security checks.

Related

How to send delete request with custom header using Spring's WebClient?

I want to send an http delete request with custom header.
I tried to do this with Spring's WebClient, with following code:
#Test
public void validateDeleteCouldCarryHeader() {
WebClient.create("https://jira.spring.io/")
.delete()
.header("X-FOO", "BAR")
.retrieve()
.bodyToMono(Map.class)
.block();
}
I am expecting something like:
DELETE / HTTP/1.1
X-FOO: BAR
...
content-length: 0
But, the actual request made is:
DELETE / HTTP/1.1
user-agent: ReactorNetty/0.7.0.RELEASE
host: jira.spring.io
accept: */*
accept-encoding: gzip
content-length: 0
Did I miss some thing to include the custom header X-FOO ?
Update 1: I upgraded reactor-netty to 0.7.3.RELEASE,
And the missing header is still missing:
DELETE / HTTP/1.1
user-agent: ReactorNetty/0.7.3.RELEASE
host: jira.spring.io
accept: */*
accept-encoding: gzip
content-length: 0
2018-12-28 22:32:32.813 DEBUG 12064 --- [ctor-http-nio-4] r.ipc.netty.http.client.HttpClient : [id: 0x2c6a9cea, L:/172.17.1.131:54532 - R:jira.spring.io/35.199.60.33:443] WRITE: 138B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 44 45 4c 45 54 45 20 2f 20 48 54 54 50 2f 31 2e |DELETE / HTTP/1.|
|00000010| 31 0d 0a 75 73 65 72 2d 61 67 65 6e 74 3a 20 52 |1..user-agent: R|
|00000020| 65 61 63 74 6f 72 4e 65 74 74 79 2f 30 2e 37 2e |eactorNetty/0.7.|
|00000030| 33 2e 52 45 4c 45 41 53 45 0d 0a 68 6f 73 74 3a |3.RELEASE..host:|
|00000040| 20 6a 69 72 61 2e 73 70 72 69 6e 67 2e 69 6f 0d | jira.spring.io.|
|00000050| 0a 61 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 61 63 |.accept: */*..ac|
|00000060| 63 65 70 74 2d 65 6e 63 6f 64 69 6e 67 3a 20 67 |cept-encoding: g|
|00000070| 7a 69 70 0d 0a 63 6f 6e 74 65 6e 74 2d 6c 65 6e |zip..content-len|
|00000080| 67 74 68 3a 20 30 0d 0a 0d 0a |gth: 0.... |
+--------+-------------------------------------------------+----------------+
Update 2: I found there's a reactor's BOM (Bismuth-RELEASE) imported in my pom, after removed that BOM, Spring Boot 2.0.5.RELEASE included the custom header I want.
DELETE / HTTP/1.1
user-agent: ReactorNetty/0.7.9.RELEASE
host: jira.spring.io
accept: */*
accept-encoding: gzip
X-FOO: BAR
Reactor Netty 0.7.0.RELEASE is quite old now, and you should upgrade to a newer version.
I've tried to reproduce this and couldn't.
Setting the log level of reactor.ipc.netty.http.client.HttpClient to "DEBUG" shows this:
[reactor-http-nio-4] DEBUG reactor.ipc.netty.http.client.HttpClient - [id: 0x69202b84, L:/192.168.0.28:60880 - R:jira.spring.io/35.199.60.33:443] WRITE: 150B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 44 45 4c 45 54 45 20 2f 20 48 54 54 50 2f 31 2e |DELETE / HTTP/1.|
|00000010| 31 0d 0a 75 73 65 72 2d 61 67 65 6e 74 3a 20 52 |1..user-agent: R|
|00000020| 65 61 63 74 6f 72 4e 65 74 74 79 2f 30 2e 37 2e |eactorNetty/0.7.|
|00000030| 30 2e 52 45 4c 45 41 53 45 0d 0a 68 6f 73 74 3a |0.RELEASE..host:|
|00000040| 20 6a 69 72 61 2e 73 70 72 69 6e 67 2e 69 6f 0d | jira.spring.io.|
|00000050| 0a 61 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 61 63 |.accept: */*..ac|
|00000060| 63 65 70 74 2d 65 6e 63 6f 64 69 6e 67 3a 20 67 |cept-encoding: g|
|00000070| 7a 69 70 0d 0a 58 2d 46 4f 4f 3a 20 42 41 52 0d |zip..X-FOO: BAR.|
|00000080| 0a 63 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 3a |.content-length:|
|00000090| 20 30 0d 0a 0d 0a | 0.... |
+--------+-------------------------------------------------+----------------+
So the client is indeed sending those headers. Maybe there's something wrong with the way you're capturing the outgoing request information?

Fine Uploader 5.11.10 SignatureDoesNotMatch with PHP & AWS S3

UPDATED POST AND RETURN INFO
I'm new to FineUploader and I'm running into problems when uploading files to AWS s3. S3 responds with <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message> I've tried many different options and parameters and have currently paired it down to a very simple configuration but still get the same response. I've followed the steps in the documentation for setting up fineuploder and the PHP signature endpoint for signing, gone through countless options/settings, looked at various posts but nothing has quite got me to a solution.
The js is simplified down to:
var s3Uploader = new qq.s3.FineUploader({
debug: true,
element: document.getElementById('fine_uploader'),
request: {
endpoint: 'https://bucketname.s3.amazonaws.com',
accessKey: 'KEY'
},
cors: {
expected: true,
sendCredentials: true
},
signature: {
endpoint: 'https://domain/s3/endpoint',
version: 4
},
uploadSuccess: {
endpoint: 'https://domain/s3/endpointsuccess'
},
iframeSupport: {
localBlankPagePath: 'https://domain/s3/blank'
},
thumbnails: {
placeholders: {
notAvailablePath: "https://domain/not_available-generic.png",
waitingPath: "https://domain/waiting-generic.png"
}
}
});
The endpoint-cors.php file for signing requests is identical to that found on Github FineUploader/php-s3-server/endpoint-cors.php dated Nov 5, 2015.
FineUploder submits the following to my endpoint for signing:
{"expiration":"2016-12-30T16:58:27.028Z","conditions":[{"acl":"private"},{"bucket":"bucketname"},{"Content-Type":"application/pdf"},{"success_action_status":"200"},{"x-amz-algorithm":"AWS4-HMAC-SHA256"},{"key":"2c6ed85e-1d5f-45e3-8458-e864f92a8cf3.pdf"},{"x-amz-credential":"KEY/20161230/us-east-1/s3/aws4_request"},{"x-amz-date":"20161230T165327Z"},{"x-amz-meta-qqfilename":"DefaultDocument.pdf"}]}
My signing endpoint responds with:
{"policy":"eyJleHBpcmF0aW9uIjoiMjAxNi0xMi0zMFQxNjo1ODoyNy4wMjhaIiwiY29uZGl0aW9ucyI6W3siYWNsIjoicHJpdmF0ZSJ9LHsiYnVja2V0IjoidXBsb2FkdG9idWlsZGVycG9wIn0seyJDb250ZW50LVR5cGUiOiJhcHBsaWNhdGlvblwvcGRmIn0seyJzdWNjZXNzX2FjdGlvbl9zdGF0dXMiOiIyMDAifSx7IngtYW16LWFsZ29yaXRobSI6IkFXUzQtSE1BQy1TSEEyNTYifSx7ImtleSI6IjJjNmVkODVlLTFkNWYtNDVlMy04NDU4LWU4NjRmOTJhOGNmMy5wZGYifSx7IngtYW16LWNyZWRlbnRpYWwiOiJBS0lBSjRaNlRWRDVVNTZOUzVRUVwvMjAxNjEyMzBcL3VzLWVhc3QtMVwvczNcL2F3czRfcmVxdWVzdCJ9LHsieC1hbXotZGF0ZSI6IjIwMTYxMjMwVDE2NTMyN1oifSx7IngtYW16LW1ldGEtcXFmaWxlbmFtZSI6IkRlZmF1bHREb2N1bWVudC5wZGYifV19","signature":"01dd1e6ab8be148c9f26b8721a267f41d0b4e1d6ac1da7ca3f5a1cc8ab9888e3"}
AWS S3 POST information for file upload:
Request URL:https://bucketname.s3.amazonaws.com/
Request Method:POST
Status Code:403 Forbidden
Remote Address:ipaddress:443
Response Headers
Access-Control-Allow-Methods:POST, PUT, DELETE
Access-Control-Allow-Origin:*
Access-Control-Expose-Headers:ETag
Access-Control-Max-Age:3000
Connection:close
Content-Type:application/xml
Date:Fri, 30 Dec 2016 16:53:29 GMT
Server:AmazonS3
Transfer-Encoding:chunked
Vary:Origin, Access-Control-Request-Headers, Access-Control-Request-Method
x-amz-id-2:lQOge5d8wTNIwH1g3TE/IuNPXNad29p6PbC8DHPDTPZzWvS9eTK6wE5nd2K6Kt/f+FsH9B0oBRw=
x-amz-request-id:E9A9C61173E13568
Request Headers
Accept:*/*
Accept-Encoding:gzip, deflate, br
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
Content-Length:659510
Content-Type:multipart/form-data; boundary=----WebKitFormBoundaryiz1gbHgrB4vnnG1m
Host:bucketname.s3.amazonaws.com
Origin:https://domain
Referer:https://domain/create/?pt=O1&pid=mFflJn
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36
Request Payload
------WebKitFormBoundaryiz1gbHgrB4vnnG1m
Content-Disposition: form-data; name="key"
2c6ed85e-1d5f-45e3-8458-e864f92a8cf3.pdf
------WebKitFormBoundaryiz1gbHgrB4vnnG1m
Content-Disposition: form-data; name="Content-Type"
application/pdf
------WebKitFormBoundaryiz1gbHgrB4vnnG1m
Content-Disposition: form-data; name="success_action_status"
200
------WebKitFormBoundaryiz1gbHgrB4vnnG1m
Content-Disposition: form-data; name="acl"
private
------WebKitFormBoundaryiz1gbHgrB4vnnG1m
Content-Disposition: form-data; name="x-amz-meta-qqfilename"
DefaultDocument.pdf
------WebKitFormBoundaryiz1gbHgrB4vnnG1m
Content-Disposition: form-data; name="x-amz-algorithm"
AWS4-HMAC-SHA256
------WebKitFormBoundaryiz1gbHgrB4vnnG1m
Content-Disposition: form-data; name="x-amz-credential"
AKIAJ4Z6TVD5U56NS5QQ/20161230/us-east-1/s3/aws4_request
------WebKitFormBoundaryiz1gbHgrB4vnnG1m
Content-Disposition: form-data; name="x-amz-date"
20161230T165327Z
------WebKitFormBoundaryiz1gbHgrB4vnnG1m
Content-Disposition: form-data; name="policy"
eyJleHBpcmF0aW9uIjoiMjAxNi0xMi0zMFQxNjo1ODoyNy4wMjhaIiwiY29uZGl0aW9ucyI6W3siYWNsIjoicHJpdmF0ZSJ9LHsiYnVja2V0IjoidXBsb2FkdG9idWlsZGVycG9wIn0seyJDb250ZW50LVR5cGUiOiJhcHBsaWNhdGlvblwvcGRmIn0seyJzdWNjZXNzX2FjdGlvbl9zdGF0dXMiOiIyMDAifSx7IngtYW16LWFsZ29yaXRobSI6IkFXUzQtSE1BQy1TSEEyNTYifSx7ImtleSI6IjJjNmVkODVlLTFkNWYtNDVlMy04NDU4LWU4NjRmOTJhOGNmMy5wZGYifSx7IngtYW16LWNyZWRlbnRpYWwiOiJBS0lBSjRaNlRWRDVVNTZOUzVRUVwvMjAxNjEyMzBcL3VzLWVhc3QtMVwvczNcL2F3czRfcmVxdWVzdCJ9LHsieC1hbXotZGF0ZSI6IjIwMTYxMjMwVDE2NTMyN1oifSx7IngtYW16LW1ldGEtcXFmaWxlbmFtZSI6IkRlZmF1bHREb2N1bWVudC5wZGYifV19
------WebKitFormBoundaryiz1gbHgrB4vnnG1m
Content-Disposition: form-data; name="x-amz-signature"
01dd1e6ab8be148c9f26b8721a267f41d0b4e1d6ac1da7ca3f5a1cc8ab9888e3
------WebKitFormBoundaryiz1gbHgrB4vnnG1m
Content-Disposition: form-data; name="file"; filename="DefaultDocument.pdf"
Content-Type: application/pdf
------WebKitFormBoundaryiz1gbHgrB4vnnG1m--
AWS S3 POST RESPONSE:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>AKIAJ4Z6TVD5U56NS5QQ</AWSAccessKeyId><StringToSign>eyJleHBpcmF0aW9uIjoiMjAxNi0xMi0zMFQxNjo1ODoyNy4wMjhaIiwiY29uZGl0aW9ucyI6W3siYWNsIjoicHJpdmF0ZSJ9LHsiYnVja2V0IjoidXBsb2FkdG9idWlsZGVycG9wIn0seyJDb250ZW50LVR5cGUiOiJhcHBsaWNhdGlvblwvcGRmIn0seyJzdWNjZXNzX2FjdGlvbl9zdGF0dXMiOiIyMDAifSx7IngtYW16LWFsZ29yaXRobSI6IkFXUzQtSE1BQy1TSEEyNTYifSx7ImtleSI6IjJjNmVkODVlLTFkNWYtNDVlMy04NDU4LWU4NjRmOTJhOGNmMy5wZGYifSx7IngtYW16LWNyZWRlbnRpYWwiOiJBS0lBSjRaNlRWRDVVNTZOUzVRUVwvMjAxNjEyMzBcL3VzLWVhc3QtMVwvczNcL2F3czRfcmVxdWVzdCJ9LHsieC1hbXotZGF0ZSI6IjIwMTYxMjMwVDE2NTMyN1oifSx7IngtYW16LW1ldGEtcXFmaWxlbmFtZSI6IkRlZmF1bHREb2N1bWVudC5wZGYifV19</StringToSign><SignatureProvided>01dd1e6ab8be148c9f26b8721a267f41d0b4e1d6ac1da7ca3f5a1cc8ab9888e3</SignatureProvided><StringToSignBytes>65 79 4a 6c 65 48 42 70 63 6d 46 30 61 57 39 75 49 6a 6f 69 4d 6a 41 78 4e 69 30 78 4d 69 30 7a 4d 46 51 78 4e 6a 6f 31 4f 44 6f 79 4e 79 34 77 4d 6a 68 61 49 69 77 69 59 32 39 75 5a 47 6c 30 61 57 39 75 63 79 49 36 57 33 73 69 59 57 4e 73 49 6a 6f 69 63 48 4a 70 64 6d 46 30 5a 53 4a 39 4c 48 73 69 59 6e 56 6a 61 32 56 30 49 6a 6f 69 64 58 42 73 62 32 46 6b 64 47 39 69 64 57 6c 73 5a 47 56 79 63 47 39 77 49 6e 30 73 65 79 4a 44 62 32 35 30 5a 57 35 30 4c 56 52 35 63 47 55 69 4f 69 4a 68 63 48 42 73 61 57 4e 68 64 47 6c 76 62 6c 77 76 63 47 52 6d 49 6e 30 73 65 79 4a 7a 64 57 4e 6a 5a 58 4e 7a 58 32 46 6a 64 47 6c 76 62 6c 39 7a 64 47 46 30 64 58 4d 69 4f 69 49 79 4d 44 41 69 66 53 78 37 49 6e 67 74 59 57 31 36 4c 57 46 73 5a 32 39 79 61 58 52 6f 62 53 49 36 49 6b 46 58 55 7a 51 74 53 45 31 42 51 79 31 54 53 45 45 79 4e 54 59 69 66 53 78 37 49 6d 74 6c 65 53 49 36 49 6a 4a 6a 4e 6d 56 6b 4f 44 56 6c 4c 54 46 6b 4e 57 59 74 4e 44 56 6c 4d 79 30 34 4e 44 55 34 4c 57 55 34 4e 6a 52 6d 4f 54 4a 68 4f 47 4e 6d 4d 79 35 77 5a 47 59 69 66 53 78 37 49 6e 67 74 59 57 31 36 4c 57 4e 79 5a 57 52 6c 62 6e 52 70 59 57 77 69 4f 69 4a 42 53 30 6c 42 53 6a 52 61 4e 6c 52 57 52 44 56 56 4e 54 5a 4f 55 7a 56 52 55 56 77 76 4d 6a 41 78 4e 6a 45 79 4d 7a 42 63 4c 33 56 7a 4c 57 56 68 63 33 51 74 4d 56 77 76 63 7a 4e 63 4c 32 46 33 63 7a 52 66 63 6d 56 78 64 57 56 7a 64 43 4a 39 4c 48 73 69 65 43 31 68 62 58 6f 74 5a 47 46 30 5a 53 49 36 49 6a 49 77 4d 54 59 78 4d 6a 4d 77 56 44 45 32 4e 54 4d 79 4e 31 6f 69 66 53 78 37 49 6e 67 74 59 57 31 36 4c 57 31 6c 64 47 45 74 63 58 46 6d 61 57 78 6c 62 6d 46 74 5a 53 49 36 49 6b 52 6c 5a 6d 46 31 62 48 52 45 62 32 4e 31 62 57 56 75 64 43 35 77 5a 47 59 69 66 56 31 39</StringToSignBytes><RequestId>E9A9C61173E13568</RequestId><HostId>lQOge5d8wTNIwH1g3TE/IuNPXNad29p6PbC8DHPDTPZzWvS9eTK6wE5nd2K6Kt/f+FsH9B0oBRw=</HostId></Error>
Comparison of the endpoint signature and the AWS S3 POST response appear the same signature and policy are being submitted.
Response Snipets:
endpoint response: {....,"signature":"01dd1e6ab8be148c9f26b8721a267f41d0b4e1d6ac1da7ca3f5a1cc8ab9888e3"}
AWS S3 response: <SignatureProvided>01dd1e6ab8be148c9f26b8721a267f41d0b4e1d6ac1da7ca3f5a1cc8ab9888e3</SignatureProvided>
endpoint policy: eyJleHBpcmF0aW9uIjoiMjAxNi0xMi0zMFQxNjo1ODoyNy4wMjhaIiwiY29uZGl0aW9ucyI6W3siYWNsIjoicHJpdmF0ZSJ9LHsiYnVja2V0IjoidXBsb2FkdG9idWlsZGVycG9wIn0seyJDb250ZW50LVR5cGUiOiJhcHBsaWNhdGlvblwvcGRmIn0seyJzdWNjZXNzX2FjdGlvbl9zdGF0dXMiOiIyMDAifSx7IngtYW16LWFsZ29yaXRobSI6IkFXUzQtSE1BQy1TSEEyNTYifSx7ImtleSI6IjJjNmVkODVlLTFkNWYtNDVlMy04NDU4LWU4NjRmOTJhOGNmMy5wZGYifSx7IngtYW16LWNyZWRlbnRpYWwiOiJBS0lBSjRaNlRWRDVVNTZOUzVRUVwvMjAxNjEyMzBcL3VzLWVhc3QtMVwvczNcL2F3czRfcmVxdWVzdCJ9LHsieC1hbXotZGF0ZSI6IjIwMTYxMjMwVDE2NTMyN1oifSx7IngtYW16LW1ldGEtcXFmaWxlbmFtZSI6IkRlZmF1bHREb2N1bWVudC5wZGYifV19
S3 Response: <StringToSign>eyJleHBpcmF0aW9uIjoiMjAxNi0xMi0zMFQxNjo1ODoyNy4wMjhaIiwiY29uZGl0aW9ucyI6W3siYWNsIjoicHJpdmF0ZSJ9LHsiYnVja2V0IjoidXBsb2FkdG9idWlsZGVycG9wIn0seyJDb250ZW50LVR5cGUiOiJhcHBsaWNhdGlvblwvcGRmIn0seyJzdWNjZXNzX2FjdGlvbl9zdGF0dXMiOiIyMDAifSx7IngtYW16LWFsZ29yaXRobSI6IkFXUzQtSE1BQy1TSEEyNTYifSx7ImtleSI6IjJjNmVkODVlLTFkNWYtNDVlMy04NDU4LWU4NjRmOTJhOGNmMy5wZGYifSx7IngtYW16LWNyZWRlbnRpYWwiOiJBS0lBSjRaNlRWRDVVNTZOUzVRUVwvMjAxNjEyMzBcL3VzLWVhc3QtMVwvczNcL2F3czRfcmVxdWVzdCJ9LHsieC1hbXotZGF0ZSI6IjIwMTYxMjMwVDE2NTMyN1oifSx7IngtYW16LW1ldGEtcXFmaWxlbmFtZSI6IkRlZmF1bHREb2N1bWVudC5wZGYifV19</StringToSign>
Thanks for any help you provide.

How to scrub VT100/ANSI control chars in Net::Telnet

I am using Net::Telnet to connect to a HP ProCurve Switch to login and backup the config. However I ran into issues where waitfor returns VT100/ANSI control chars:
< 0x00000: ff fd 18 ff fd 1f ff fb 01 1b 5b 32 4a 1b 5b 3f ..........[2J.[?
< 0x00010: 37 6c 1b 5b 33 3b 32 33 72 1b 5b 3f 36 6c 1b 5b 7l.[3;23r.[?6l.[
< 0x00020: 31 3b 31 48 1b 5b 3f 32 35 6c 1b 5b 31 3b 31 48 1;1H.[?25l.[1;1H
< 0x00030: 48 50 20 4a 39 37 32 38 41 20 32 39 32 30 2d 34 HP J9728A 2920-4
< 0x00040: 38 47 20 53 77 69 74 63 68 0d 0d 0a 53 6f 66 74 8G Switch...Soft
< 0x00050: 77 61 72 65 20 72 65 76 69 73 69 6f 6e 20 57 42 ware revision WB
< 0x00060: 2e 31 35 2e 31 32 2e 30 30 31 35 0d 0d 0a 0d 0d .15.12.0015.....
< 0x00070: 0a 43 6f 70 79 72 69 67 68 74 20 28 43 29 20 31 .Copyright (C) 1
< 0x00080: 39 39 31 2d 32 30 31 34 20 48 65 77 6c 65 74 74 991-2014 Hewlett
< 0x00090: 2d 50 61 63 6b 61 72 64 20 44 65 76 65 6c 6f 70 -Packard Develop
< 0x000a0: 6d 65 6e 74 20 43 6f 6d 70 61 6e 79 2c 20 4c 2e ment Company, L.
< 0x000b0: 50 2e 0d 0a 0d 0a 20 20 20 20 20 20 20 20 20 20 P.....
Unfortunately, this screws up waitfor because if I try to waitfor(/^password:/i) it will return a string with those control chars in it, or wait forever since the regex is never matched.
Is there any way to have Net::Telnet automatically remove those control characters? Is there any way to have waitfor only care about ASCII printable characters?

JPOS Q2 : Unable to see raw ISO 8583 data

[Update]
I was able to bring up the JPOS client and server simulator on the same box using this link : http://jpos.org/blog/2013/07/setting-up-the-client-simulator/( Please note the setup is pretty similar to one described in the link for running a server simulator too).
What i did next was to basically try to see the tcpdump ( also using wireshark). But what i see is not what i expected. Here's what i see ( Please note the data part)
Data (325 bytes)
0000 3c 69 73 6f 6d 73 67 3e 0a 20 20 3c 21 2d 2d 20 <isomsg>. <!--
0010 6f 72 67 2e 6a 70 6f 73 2e 69 73 6f 2e 70 61 63 org.jpos.iso.pac
0020 6b 61 67 65 72 2e 58 4d 4c 50 61 63 6b 61 67 65 kager.XMLPackage
0030 72 20 2d 2d 3e 0a 20 20 3c 66 69 65 6c 64 20 69 r -->. <field i
0040 64 3d 22 30 22 20 76 61 6c 75 65 3d 22 31 38 30 d="0" value="180
0050 30 22 2f 3e 0a 20 20 3c 66 69 65 6c 64 20 69 64 0"/>. <field id
0060 3d 22 37 22 20 76 61 6c 75 65 3d 22 30 37 32 30 ="7" value="0720
0070 30 30 33 36 33 39 22 2f 3e 0a 20 20 3c 66 69 65 003639"/>. <fie
0080 6c 64 20 69 64 3d 22 31 31 22 20 76 61 6c 75 65 ld id="11" value
0090 3d 22 37 39 39 38 31 33 22 2f 3e 0a 20 20 3c 66 ="799813"/>. <f
00a0 69 65 6c 64 20 69 64 3d 22 31 32 22 20 76 61 6c ield id="12" val
00b0 75 65 3d 22 37 39 39 38 30 35 22 2f 3e 0a 20 20 ue="799805"/>.
00c0 3c 66 69 65 6c 64 20 69 64 3d 22 36 33 22 20 76 <field id="63" v
00d0 61 6c 75 65 3d 22 4d 6f 6e 20 4a 75 6c 20 32 30 alue="Mon Jul 20
00e0 20 30 30 3a 33 36 3a 33 39 20 50 44 54 20 32 30 00:36:39 PDT 20
00f0 31 35 22 2f 3e 0a 20 20 3c 69 73 6f 6d 73 67 20 15"/>. <isomsg
0100 69 64 3d 22 31 32 30 22 3e 0a 20 20 20 20 3c 66 id="120">. <f
0110 69 65 6c 64 20 69 64 3d 22 30 22 20 76 61 6c 75 ield id="0" valu
0120 65 3d 22 32 39 31 31 30 30 30 31 22 2f 3e 0a 20 e="29110001"/>.
0130 20 3c 2f 69 73 6f 6d 73 67 3e 0a 3c 2f 69 73 6f </isomsg>.</iso
0140 6d 73 67 3e 0a msg>.
Data: 3c69736f6d73673e0a20203c212d2d206f72672e6a706f73...
[Length: 325]
If you look at the data, it looks like the XML ISO Msg. I was expecting something like the HEX representation of ISO 8583 where the first bytes are the MTI and etc etc..
After looking at the client simulator file, i realized that its a XML Channel and packager. I looked at the following channel & packager link here jpos.org/doc/javadoc/org/jpos/iso/packager/package-summary.html jpos.org/doc/javadoc/org/jpos/iso/channel/package-summary.html
After changing the packager to PostChannel and PostPackager, i still see the problems on my client and i see it times out. Was wondering if there was a way to see the actual raw data via tcpdump/wireshark. The most close is the Postilion which has data length prepended to the raw data.
After playing with the PostChannel and PostPackager, i was able to get it running and could see the message. The things i needed to do was basically change both the server simulator and client simulator configurations to use the desired Channel and Packager.
This is what i changed in both the server and client simulator
Server Simulator : Change the file src/dist/deploy/05_serversimulator.xml to use the desired channel and packager
<channel class="org.jpos.iso.channel.PostChannel" logger="Q2"
packager="org.jpos.iso.packager.PostPackager">
Client Simulator : Change the file ./src/dist/deploy/10_clientsimulator_channel.xml to use the desired channel and packager
<channel class="org.jpos.iso.channel.PostChannel" logger="Q2"
packager="org.jpos.iso.packager.PostPackager">
And then fire up the client and server simulators.
Channels assist you in connecting to the other entity and add headers, length headers , tpdu etc based on the implementation of the channel used.
PostChannel that you use here adds a 2 byte length header containing the size of the message. This assists the receiver in collecting the right amount of bytes from the tcp stream.
Packagers assist you in packing fields in the message, examples are fixed field, length prepended variables fields and what encoding these should have (hex,bcd, ascii).
The client server sims out of the box use xml for understanding the concepts.

How do I connect to a websocket manually, with netcat/socat/telnet?

I am trying to connect to the reference websocket echo server "manually", in order to learn how the protocol works (I am using socat for that). However, the server invariably closes the connection without providing an answer. Any idea why?
Here is what I do:
socat - TCP:echo.websocket.org:80
Then, I paste the following text in the terminal:
GET /?encoding=text HTTP/1.1
Origin: http://www.websocket.org
Connection: Upgrade
Host: echo.websocket.org
Sec-WebSocket-Key: P7Kp2hTLNRPFMGLxPV47eQ==
Upgrade: websocket
Sec-WebSocket-Version: 13
I sniffed the parameters of the connection with the developer tools, in firefox, on the same machine, where this works flawlessly: therefore, I would assume they are correct. However after that, the server closes the connection immediately, without providing an answer. Why? How can I implement the protocol "manually"?
I would like type test in my terminal and get the server to reply with what I typed (It works in a web browser).
I think you want to modify the socket stream to translate \n (line feed) to CRLF (Carriage return & line feed). Doing info socat produces detailed information which includes this modifier:
crnl Converts the default line termination character NL ('\n', 0x0a)
to/from CRNL ("\r\n", 0x0d0a) when writing/reading on this chan-
nel (example). Note: socat simply strips all CR characters.
So I think you should be able to do this:
socat - TCP:echo.websocket.org:80,crnl
I'd like to add that my WebSocket tool websocat can help in debugging the WebSocket protocol, especially when combined with socat:
$ websocat - ws-c:sh-c:"socat -v -x - tcp:echo.websocket.org:80" --ws-c-uri ws://echo.websocket.org
> 2018/07/03 16:30:06.021658 length=157 from=0 to=156
47 45 54 20 2f 20 48 54 54 50 2f 31 2e 31 0d 0a GET / HTTP/1.1..
48 6f 73 74 3a 20 65 63 68 6f 2e 77 65 62 73 6f Host: echo.webso
63 6b 65 74 2e 6f 72 67 0d 0a cket.org..
43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 55 70 67 72 Connection: Upgr
61 64 65 0d 0a ade..
55 70 67 72 61 64 65 3a 20 77 65 62 73 6f 63 6b Upgrade: websock
65 74 0d 0a et..
53 65 63 2d 57 65 62 53 6f 63 6b 65 74 2d 56 65 Sec-WebSocket-Ve
72 73 69 6f 6e 3a 20 31 33 0d 0a rsion: 13..
53 65 63 2d 57 65 62 53 6f 63 6b 65 74 2d 4b 65 Sec-WebSocket-Ke
79 3a 20 59 76 36 32 44 31 57 6d 7a 79 79 31 65 y: Yv62D1Wmzyy1e
69 6d 62 47 6d 68 69 61 67 3d 3d 0d 0a imbGmhiag==..
0d 0a ..
--
< 2018/07/03 16:30:06.164057 length=201 from=0 to=200
48 54 54 50 2f 31 2e 31 20 31 30 31 20 57 65 62 HTTP/1.1 101 Web
20 53 6f 63 6b 65 74 20 50 72 6f 74 6f 63 6f 6c Socket Protocol
20 48 61 6e 64 73 68 61 6b 65 0d 0a Handshake..
43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 55 70 67 72 Connection: Upgr
61 64 65 0d 0a ade..
44 61 74 65 3a 20 54 75 65 2c 20 30 33 20 4a 75 Date: Tue, 03 Ju
6c 20 32 30 31 38 20 31 33 3a 31 35 3a 30 30 20 l 2018 13:15:00
47 4d 54 0d 0a GMT..
53 65 63 2d 57 65 62 53 6f 63 6b 65 74 2d 41 63 Sec-WebSocket-Ac
63 65 70 74 3a 20 55 56 6a 32 74 35 50 43 7a 62 cept: UVj2t5PCzb
58 49 32 52 4e 51 75 70 2f 71 48 31 63 5a 44 6e XI2RNQup/qH1cZDn
38 3d 0d 0a 8=..
53 65 72 76 65 72 3a 20 4b 61 61 7a 69 6e 67 20 Server: Kaazing
47 61 74 65 77 61 79 0d 0a Gateway..
55 70 67 72 61 64 65 3a 20 77 65 62 73 6f 63 6b Upgrade: websock
65 74 0d 0a et..
0d 0a ..
--
ABCDEF
> 2018/07/03 16:30:12.707919 length=13 from=157 to=169
82 87 40 57 f5 88 01 15 b6 cc 05 11 ff ..#W.........
--
< 2018/07/03 16:30:12.848398 length=9 from=201 to=209
82 07 41 42 43 44 45 46 0a ..ABCDEF.
--
ABCDEF
> 2018/07/03 16:30:14.528333 length=6 from=170 to=175
88 80 18 ec 05 a8 ......
--
< 2018/07/03 16:30:14.671629 length=2 from=210 to=211
88 00 ..
--
In case of failures with manually driven socat -v -x - TCP:echo.websocket.org:80,crnl (mentioned in the other answer), you can compare it with WebSocat-driven socat like in session depicted above.
Reverse (server) example with socat debug dump:
socat -v -x tcp-l:1234,fork,reuseaddr exec:'websocat -t ws-u\:stdio\: mirror\:'
Alternatively, here is a way to connect and read the stream from a wss secure websocket stream from the command line using solely core php.
php -r '$sock=stream_socket_client("tls://echo.websocket.org:443",$e,$n,30,STREAM_CLIENT_CONNECT,stream_context_create(null));if(!$sock){echo"[$n]$e".PHP_EOL;}else{fwrite($sock,"GET / HTTP/1.1\r\nHost: echo.websocket.org\r\nAccept: */*\r\nConnection: Upgrade\r\nUpgrade: websocket\r\nSec-WebSocket-Version: 13\r\nSec-WebSocket-Key: ".rand(0,999)."\r\n\r\n");while(!feof($sock)){var_dump(fgets($sock,2048));}}'
Other similar example, pulling from another wss server: (Do not get rekt)
php -r '$sock=stream_socket_client("tls://stream.binance.com:9443",$e,$n,30,STREAM_CLIENT_CONNECT,stream_context_create(null));if(!$sock){echo"[$n]$e".PHP_EOL;}else{fwrite($sock,"GET /stream?streams=btcusdt#kline_1m HTTP/1.1\r\nHost: stream.binance.com:9443\r\nAccept: */*\r\nConnection: Upgrade\r\nUpgrade: websocket\r\nSec-WebSocket-Version: 13\r\nSec-WebSocket-Key: ".rand(0,999)."\r\n\r\n");while(!feof($sock)){var_dump(explode(",",fgets($sock,512)));}}'

Resources