Signature Mismatch error when using S3 Signed URL - laravel

I don't know what happened in the last 24 hours that caused this thing to break, but it had been working perfectly for 3 years, non stop without issues. Trust me, I haven't changed any code. As a matter of fact, I haven't pushed code for 2 weeks.
Basically, I am using Laravel 6 and Laravel Vapor to upload files to S3 using the Signed URLs mechanism that the framework provides. Ever since this morning, I am seeing the following issue. The AWS PHP SDK is 3.148.3, by the way.
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
<AWSAccessKeyId>[REDACTED]</AWSAccessKeyId>
<StringToSign>AWS4-HMAC-SHA256 20220712T203731Z 20220712/eu-west-2/s3/aws4_request 961d5d32676be187af1a7ccf8a92b8c29e2d851bdd51c46836aedce7cc9e089a</StringToSign>
<SignatureProvided>101e80ad27bfe4ed12a34b19fff4c17f87a3f639b92d62c8e7a79b13c876b1ca</SignatureProvided>
<StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 32 30 37 31 32 54 32 30 33 37 33 31 5a 0a 32 30 32 32 30 37 31 32 2f 65 75 2d 77 65 73 74 2d 32 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 39 36 31 64 35 64 33 32 36 37 36 62 65 31 38 37 61 66 31 61 37 63 63 66 38 61 39 32 62 38 63 32 39 65 32 64 38 35 31 62 64 64 35 31 63 34 36 38 33 36 61 65 64 63 65 37 63 63 39 65 30 38 39 61</StringToSignBytes>
<CanonicalRequest>GET /tmp/0344aa31-c756-49c3-8ce1-932ec23c895d X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=[REDACTED]%2F20220712%2Feu-west-2%2Fs3%2Faws4_request&X-Amz-Date=20220712T203731Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host%3Bx-amz-acl&x-amz-acl=public-read host:[REDACTED].s3.eu-west-2.amazonaws.com x-amz-acl:public-read host;x-amz-acl UNSIGNED-PAYLOAD</CanonicalRequest>
<CanonicalRequestBytes>47 45 54 0a 2f 74 6d 70 2f 30 33 34 34 61 61 33 31 2d 63 37 35 36 2d 34 39 63 33 2d 38 63 65 31 2d 39 33 32 65 63 32 33 63 38 39 35 64 0a 58 2d 41 6d 7a 2d 41 6c 67 6f 72 69 74 68 6d 3d 41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 26 58 2d 41 6d 7a 2d 43 6f 6e 74 65 6e 74 2d 53 68 61 32 35 36 3d 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44 26 58 2d 41 6d 7a 2d 43 72 65 64 65 6e 74 69 61 6c 3d 41 4b 49 41 34 58 4b 4a 50 55 57 34 55 43 42 4a 49 32 57 57 25 32 46 32 30 32 32 30 37 31 32 25 32 46 65 75 2d 77 65 73 74 2d 32 25 32 46 73 33 25 32 46 61 77 73 34 5f 72 65 71 75 65 73 74 26 58 2d 41 6d 7a 2d 44 61 74 65 3d 32 30 32 32 30 37 31 32 54 32 30 33 37 33 31 5a 26 58 2d 41 6d 7a 2d 45 78 70 69 72 65 73 3d 33 30 30 26 58 2d 41 6d 7a 2d 53 69 67 6e 65 64 48 65 61 64 65 72 73 3d 68 6f 73 74 25 33 42 78 2d 61 6d 7a 2d 61 63 6c 26 78 2d 61 6d 7a 2d 61 63 6c 3d 70 75 62 6c 69 63 2d 72 65 61 64 0a 68 6f 73 74 3a 73 68 61 64 6f 77 66 6f 75 6e 64 72 2d 75 70 6c 6f 61 64 73 2d 6c 6f 63 61 6c 2e 73 33 2e 65 75 2d 77 65 73 74 2d 32 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 78 2d 61 6d 7a 2d 61 63 6c 3a 70 75 62 6c 69 63 2d 72 65 61 64 0a 0a 68 6f 73 74 3b 78 2d 61 6d 7a 2d 61 63 6c 0a 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44</CanonicalRequestBytes>
<RequestId>S6E8RFNVQ3WGAAHB</RequestId>
<HostId>yA998rWcqzfVIdD9HyrIJRJtbI/ge5opCEHEgpPoBBAuwRjgABub590OnA66JVHwyxHdo/a2uB6+ii/IxwZIGw==</HostId>
</Error>
Vue.js
#change=upload
convertFileNameToKebabCase(fileNameRaw) {
let fileNameActual = _.kebabCase(fileNameRaw.split(".").shift());
let fileExtension = fileNameRaw.split(".").pop();
return `${fileNameActual}.${fileExtension}`;
},
async store(file, options = {}, fileAcl) {
const response = await Vue.axios.post(
"/vapor/signed-storage-url", {
bucket: options.bucket || "",
content_type: options.contentType || file.type,
expires: options.expires || "",
visibility: fileAcl || "",
ignore_web_middleware: true
}, {
baseURL: options.baseURL || null,
headers: options.headers || {}
}
);
let headers = response.data.headers;
if ("Host" in headers) {
delete headers.Host;
}
if (typeof options.progress === "undefined") {
options.progress = () => {};
}
// Reset axios instance as Amazon
// does not like our Auth header
const newInstance = axios.create();
await newInstance.put(response.data.url, file, {
headers: headers,
onUploadProgress: progressEvent => {
options.progress(
progressEvent.loaded / progressEvent.total
);
}
});
response.data.extension = file.name.split(".").pop();
return response.data;
},
async upload(event) {
const {
valid
} = await this.$refs.provider.validate(event);
if (valid) {
this.reset();
let fileData = this.$refs.file.files[0];
this.fileName = fileData.name;
this.url = false;
try {
const response = await this.store(
this.$refs.file.files[0], {
progress: progress => {
this.uploadProgress = Math.round(progress * 100);
}
},
"public-read"
);
Vue.axios
.post("/file/commit", {
uuid: response.uuid,
key: response.key,
bucket: response.bucket,
name: this.convertFileNameToKebabCase(this.$refs.file.files[0].name),
content_type: this.$refs.file.files[0].type
})
.then(success => {
this.reset();
this.url = success.data.data.file;
this.uploadCompleted = true;
})
.catch(error => {
this.reset();
this.uploadFailed = true;
});
} catch (error) {
this.reset();
this.uploadFailed = true;
}
}
},

Three things to check:
Region mismatch, the bucket is in region X but CLI/SDK goes to Region Y.
A corruption or unusual setting in your AWS CLI profile, reset the
profile, check if access key and secret are still valid.
CORS configuration not containing <AllowedHeader>*</AllowedHeader>
Otherwise could you share some more details on the code that causes the error?

I found the solution. I had to remove the following in the headers: x-amz-acl. I do not know why this was the problem but it seems to have worked. Hope if anyone comes across this in the future, try this solution.
if ("Host" in headers) {
delete headers.Host;
delete headers["x-amz-acl"];
}

Related

How to read a hexadecimal file and convert the content to byte slice in golang?

The hexadecimal file is from the charles proxy's hex format, and it may contains invisible characters, the example format content is:
00000000 7b 22 73 75 70 70 6f 72 74 73 5f 69 6d 70 6c 69 {"supports_impli
00000010 63 69 74 5f 73 64 6b 5f 6c 6f 67 67 69 6e 67 22 cit_sdk_logging"
00000020 3a 74 72 75 65 2c 22 67 64 70 76 34 5f 6e 75 78 :true,"gdpv4_nux
00000030 5f 65 6e 61 62 6c 65 64 22 3a 66 61 6c 73 65 2c _enabled":false,
00000040 22 61 6e 64 72 6f 69 64 5f 73 64 6b 5f 65 72 72 "android_sdk_err
00000050 6f 72 5f 63 61 74 65 67 6f 72 69 65 73 22 3a 5b or_categories":[
00000060 7b 22 6e 61 6d 65 22 3a 22 6c 6f 67 69 6e 5f 72 {"name":"login_r
00000070 65 63 6f 76 65 72 61 62 6c 65 22 2c 22 69 74 65 ecoverable","ite
00000080 6d 73 22 3a 5b 7b 22 63 6f 64 65 22 3a 31 30 32 ms":[{"code":102
00000090 7d 2c 7b 22 63 6f 64 65 22 3a 31 39 30 7d 5d 2c },{"code":190}],
000000a0 22 72 65 63 6f 76 65 72 79 5f 6d 65 73 73 61 67 "recovery_messag
000000b0 65 22 3a 22 5c 75 38 62 66 37 5c 75 39 31 63 64 e":"\u8bf7\u91cd
000000c0 5c 75 36 35 62 30 5c 75 37 36 37 62 5c 75 35 66 \u65b0\u767b\u5f
000000d0 35 35 5c 75 35 65 39 34 5c 75 37 35 32 38 5c 75 55\u5e94\u7528\u
000000e0 37 61 30 62 5c 75 35 65 38 66 5c 75 66 66 30 63 7a0b\u5e8f\uff0c
000000f0 5c 75 35 31 38 64 5c 75 36 62 32 31 5c 75 38 66 \u518d\u6b21\u8f
00000100 64 65 5c 75 36 33 61 35 20 46 61 63 65 62 6f 6f de\u63a5 Faceboo
00000110 6b 20 5c 75 35 65 31 30 5c 75 36 32 33 37 5c 75 k \u5e10\u6237\u
00000120 33 30 30 32 22 7d 5d 2c 22 61 70 70 5f 65 76 65 3002"}],"app_eve
00000130 6e 74 73 5f 73 65 73 73 69 6f 6e 5f 74 69 6d 65 nts_session_time
00000140 6f 75 74 22 3a 36 30 2c 22 61 70 70 5f 65 76 65 out":60,"app_eve
00000150 6e 74 73 5f 66 65 61 74 75 72 65 5f 62 69 74 6d nts_feature_bitm
00000160 61 73 6b 22 3a 36 35 35 35 39 2c 22 73 65 61 6d ask":65559,"seam
00000170 6c 65 73 73 5f 6c 6f 67 69 6e 22 3a 31 2c 22 73 less_login":1,"s
00000180 6d 61 72 74 5f 6c 6f 67 69 6e 5f 62 6f 6f 6b 6d mart_login_bookm
00000190 61 72 6b 5f 69 63 6f 6e 5f 75 72 6c 22 3a 22 68 ark_icon_url":"h
000001a0 74 74 70 73 3a 5c 2f 5c 2f 73 74 61 74 69 63 2e ttps:\/\/static.
000001b0 78 78 2e 66 62 63 64 6e 2e 6e 65 74 5c 2f 72 73 xx.fbcdn.net\/rs
000001c0 72 63 2e 70 68 70 5c 2f 76 33 5c 2f 79 73 5c 2f rc.php\/v3\/ys\/
000001d0 72 5c 2f 43 36 5a 75 74 59 44 53 61 61 56 2e 70 r\/C6ZutYDSaaV.p
000001e0 6e 67 22 2c 22 73 6d 61 72 74 5f 6c 6f 67 69 6e ng","smart_login
000001f0 5f 6d 65 6e 75 5f 69 63 6f 6e 5f 75 72 6c 22 3a _menu_icon_url":
00000200 22 68 74 74 70 73 3a 5c 2f 5c 2f 73 74 61 74 69 "https:\/\/stati
00000210 63 2e 78 78 2e 66 62 63 64 6e 2e 6e 65 74 5c 2f c.xx.fbcdn.net\/
00000220 72 73 72 63 2e 70 68 70 5c 2f 76 33 5c 2f 79 73 rsrc.php\/v3\/ys
00000230 5c 2f 72 5c 2f 30 69 61 72 70 6e 77 64 6d 45 78 \/r\/0iarpnwdmEx
00000240 2e 70 6e 67 22 2c 22 72 65 73 74 72 69 63 74 69 .png","restricti
00000250 76 65 5f 64 61 74 61 5f 66 69 6c 74 65 72 5f 70 ve_data_filter_p
00000260 61 72 61 6d 73 22 3a 22 7b 7d 22 2c 22 61 61 6d arams":"{}","aam
00000270 5f 72 75 6c 65 73 22 3a 22 7b 7d 22 2c 22 73 75 _rules":"{}","su
00000280 67 67 65 73 74 65 64 5f 65 76 65 6e 74 73 5f 73 ggested_events_s
00000290 65 74 74 69 6e 67 22 3a 22 7b 5c 22 70 72 6f 64 etting":"{\"prod
000002a0 75 63 74 69 6f 6e 5f 65 76 65 6e 74 73 5c 22 3a uction_events\":
000002b0 5b 5d 2c 5c 22 65 6c 69 67 69 62 6c 65 5f 66 6f [],\"eligible_fo
000002c0 72 5f 70 72 65 64 69 63 74 69 6f 6e 5f 65 76 65 r_prediction_eve
000002d0 6e 74 73 5c 22 3a 5b 5c 22 66 62 5f 6d 6f 62 69 nts\":[\"fb_mobi
000002e0 6c 65 5f 61 64 64 5f 74 6f 5f 63 61 72 74 5c 22 le_add_to_cart\"
000002f0 2c 5c 22 66 62 5f 6d 6f 62 69 6c 65 5f 70 75 72 ,\"fb_mobile_pur
00000300 63 68 61 73 65 5c 22 2c 5c 22 66 62 5f 6d 6f 62 chase\",\"fb_mob
00000310 69 6c 65 5f 63 6f 6d 70 6c 65 74 65 5f 72 65 67 ile_complete_reg
00000320 69 73 74 72 61 74 69 6f 6e 5c 22 2c 5c 22 66 62 istration\",\"fb
00000330 5f 6d 6f 62 69 6c 65 5f 69 6e 69 74 69 61 74 65 _mobile_initiate
00000340 64 5f 63 68 65 63 6b 6f 75 74 5c 22 5d 7d 22 2c d_checkout\"]}",
00000350 22 69 64 22 3a 22 31 36 33 35 34 33 35 31 34 39 "id":"1635435149
00000360 30 39 30 34 35 22 7d 09045"}
Read the file and convert the hexadecimal content to golang slice []byte, is there any convenient method? how to split the lines in the file?
I have solved it by using fixed length of the hexadecimal content. the offset length of each line is 8, so the hexadecimal content begins at offset 8, and the max length of the hexadecimal is 57. The code is as follows:
const (
beginOffset = 8 // the offset of the hexadecimal content begins
endLength = 57 // the length of the hexadecimal content ends
)
func ReadHexData(src []byte) ([]byte, error) {
lines := strings.Split(string(src), "\n")
var data string
for _, line := range lines {
for i := beginOffset; i < endLength; i++ {
if line[i] != ' ' {
data = data + string(line[i])
}
}
}
return hex.DecodeString(data)
}

nifi convert text file to json

I'm trying to load log text files from a ftp server to elastic .
The log files look like this :
0:0:21: Processing events from events
0:0:21: Processing croned build types from q_type
0:0:21: Process croned releases from trls
0:0:22: Processing croned regression list from regression
0:0:22: Processing commit loop
in data provenance (hex view , because other views not showing anything)i see the data like this :
0x00000090 66 69 65 6C 64 3A 20 52 4E 20 53 74 61 74 75 73 field: RN Status
0x000000A0 2E 20 4F 62 6A 65 63 74 20 72 65 66 65 72 65 6E . Object referen
0x000000B0 63 65 20 6E 6F 74 20 73 65 74 20 74 6F 20 61 6E ce not set to an
0x000000C0 20 69 6E 73 74 61 6E 63 65 20 6F 66 20 61 6E 20 instance of an
0x000000D0 6F 62 6A 65 63 74 2E 0D 0A 30 3A 30 3A 31 34 3A object...0:0:14:
0x000000E0 20 43 61 6E 27 74 20 72 65 61 64 20 69 73 73 75 Can't read issu
0x000000F0 65 3A 20 41 49 2D 32 34 37 20 63 75 73 74 6F 6D e: AI-247 custom
0x00000100 20 66 69 65 6C 64 3A 20 52 4E 20 53 65 63 74 69 field: RN Secti
0x00000110 6F 6E 2E 20 4F 62 6A 65 63 74 20 72 65 66 65 72 on. Object refer
0x00000120 65 6E 63 65 20 6E 6F 74 20 73 65 74 20 74 6F 20 ence not set to
0x00000130 61 6E 20 69 6E 73 74 61 6E 63 65 20 6F 66 20 61 an instance of a
0x00000140 6E 20 6F 62 6A 65 63 74 2E 0D 0A 30 3A 30 3A 31 n object...0:0:1
0x00000150 34 3A 20 43 61 6E 27 74 20 72 65 61 64 20 69 73 4: Can't read is
0x00000160 73 75 65 3A 20 41 49 2D 32 34 37 20 63 75 73 74 sue: AI-247 cust
0x00000170 6F 6D 20 66 69 65 6C 64 3A 20 52 4E 20 44 6F 63 om field: RN Doc
0x00000180 20 69 6E 20 56 65 72 2E 20 4F 62 6A 65 63 74 20 in Ver. Object
0x00000190 72 65 66 65 72 65 6E 63 65 20 6E 6F 74 20 73 65 reference not se
0x000001A0 74 20 74 6F 20 61 6E 20 69 6E 73 74 61 6E 63 65 t to an instance
0x000001B0 20 6F 66 20 61 6E 20 6F 62 6A 65 63 74 2E 0D 0A of an object...
0x000001C0 30 3A 30 3A 31 34 3A 20 43 61 6E 27 74 20 72 65 0:0:14: Can't re
0x000001D0 61 64 20 69 73 73 75 65 3A 20 41 49 2D 32 34 37 ad issue: AI-247
I can get the file with "getftp" processor, but how do I convert it to json so I can send it to Elastic ?
I am new to nifi hope im not missing something basic, any help will be appreciated.
Thanks
You can use the ConvertRecord processor with a CSVReader for the input (configure to use : as the delimiter) and a JsonRecordSetWriter for the output.
NiFi can automatically infer the schema, but as it doesn't appear you have a header line for the incoming data, this will probably not be helpful. In that case, you can use the Schema Registry to hold two schemas -- one for the incoming log lines, indicating what each field should be called and the data type, and one for the JSON output. Bryan Bende has written a great article about this process.

Replacing string with special symbol in a csv file sed

I have a csv file /tmp/test.csv with the following content.
VM,Datacenter,Cluster,Host,Folder,OS,VM ID,VM UUID,vCenter UUID
A0F0US014XVM022,"/AMMWDC04_DC/A0F0 <96> AA DR","Red Hat Enterprise Linux 6 (64-bit)",vm-2910,421f2eba-8b60-6166-3b56-f22e3f71eecf,94694731-df3a-4ee6-9962-49df97a6f08d
I want to replace <96> (surrounded by space) with - in the csv file. I tried sed -i -e 's/<96>/-/g' /tmp/test.csv but this did not work. May be because of the special symbols involved.
sed version 4.2.1
[root#fmsprdchef001 ~]# grep vm-2910 /tmp/test.csv | hexdump -C
00000000 41 30 46 30 55 53 30 31 34 58 56 4d 30 32 32 2c |A0F0US014XVM022,|
000000e0 41 4d 4d 57 44 43 30 34 5f 44 43 2f 41 30 46 30 |AMMWDC04_DC/A0F0|
000000f0 20 96 20 41 41 20 44 52 22 2c 22 52 65 64 20 48 | . AA DR","Red H|
00000100 61 74 20 45 6e 74 65 72 70 72 69 73 65 20 4c 69 |at Enterprise Li|
00000110 6e 75 78 20 36 20 28 36 34 2d 62 69 74 29 22 2c |nux 6 (64-bit)",|
00000120 76 6d 2d 32 39 31 30 2c 34 32 31 66 32 65 62 61 |vm-2910,421f2eba|
00000130 2d 38 62 36 30 2d 36 31 36 36 2d 33 62 35 36 2d |-8b60-6166-3b56-|
00000140 66 32 32 65 33 66 37 31 65 65 63 66 2c 39 34 36 |f22e3f71eecf,946|
00000150 39 34 37 33 31 2d 64 66 33 61 2d 34 65 65 36 2d |94731-df3a-4ee6-|
00000160 39 34 36 32 2d 34 39 64 66 39 37 61 36 66 30 38 |9462-49df97a6f08|
00000170 64 0a 41 30 46 30 55 53 30 31 34 58 56 4d 30 32 |d.A0F0US014XVM02|
00000180 32 2c 70 6f 77 65 72 65 64 4f 6e 2c 46 61 6c 73 |2,poweredOn,Fals|
00000190 65 2c 56 6d 78 6e 65 74 33 2c 2c 2c 54 72 75 65 |e,Vmxnet3,,,True|
000001a0 2c 54 72 75 65 2c 30 30 3a 35 30 3a 35 36 3a 39 |,True,00:50:56:9|
000001b0 66 3a 30 31 3a 62 38 2c 61 73 73 69 67 6e 65 64 |f:01:b8,assigned|
000001c0 2c 22 31 30 2e 31 30 30 2e 31 2e 31 32 2c 20 66 |,"10.100.1.12, f|
000001d0 65 38 30 3a 3a 32 35 30 3a 35 36 66 66 3a 66 65 |e80::250:56ff:fe|
000001e0 39 66 3a 31 62 38 22 2c 22 41 6d 65 72 69 63 61 |9f:1b8","America|
000001f0 6e 20 41 69 72 6c 69 6e 65 73 3b 20 50 52 44 3b |n Airlines; PRD;|
00000200 20 61 70 70 20 26 20 44 42 32 3b 44 52 22 2c 41 | app & DB2;DR",A|
00000210 4d 4d 57 44 43 30 34 5f 44 43 2c 41 4d 4d 57 44 |MMWDC04_DC,AMMWD|
00000220 43 30 34 43 41 2c 61 6d 6d 77 64 63 30 34 63 75 |C04CA,ammwdc04cu|
00000230 73 74 65 73 78 30 31 2e 69 6d 7a 63 6c 6f 75 64 |stesx01.imzcloud|
00000240 2e 69 62 6d 61 6d 6d 73 61 70 2e 6c 6f 63 61 6c |.ibmammsap.local|
00000250 2c 22 2f 41 4d 4d 57 44 43 30 34 5f 44 43 2f 41 |,"/AMMWDC04_DC/A|
00000260 30 46 30 20 96 20 41 41 20 44 52 22 2c 22 52 65 |0F0 . AA DR","Re|
00000270 64 20 48 61 74 20 45 6e 74 65 72 70 72 69 73 65 |d Hat Enterprise|
00000280 20 4c 69 6e 75 78 20 36 20 28 36 34 2d 62 69 74 | Linux 6 (64-bit|
00000290 29 22 2c 76 6d 2d 32 39 31 30 2c 34 32 31 66 32 |)",vm-2910,421f2|
000002a0 65 62 61 2d 38 62 36 30 2d 36 31 36 36 2d 33 62 |eba-8b60-6166-3b|
000002b0 35 36 2d 66 32 32 65 33 66 37 31 65 65 63 66 2c |56-f22e3f71eecf,|
000002c0 39 34 36 39 34 37 33 31 2d 64 66 33 61 2d 34 65 |94694731-df3a-4e|
000002d0 65 36 2d 39 34 36 32 2d 34 39 64 66 39 37 61 36 |e6-9462-49df97a6|
000002e0 66 30 38 64 0a |f08d.|
000002e5
This is the important part from your hexdump:
000000f0 20 96 20 41 41 20 44 52 22 2c 22 52 65 64 20 48 | . AA DR","Red H|
I suggest:
sed -i 's/ \x96 /-/' file
Following sed may help you on same.
sed 's/\([^<]*\)\(<.*>\)\(.*\)/\1-\3/' Input_file
In case you want to save the output into same Input_file then use -i option with sed in above code.
EDIT: In case your Input_file is having control characters then use following.
sed 's/\r//g;s/\([^<]*\)\(<.*>\)\(.*\)/\1-\3/' Input_file

How to scrub VT100/ANSI control chars in Net::Telnet

I am using Net::Telnet to connect to a HP ProCurve Switch to login and backup the config. However I ran into issues where waitfor returns VT100/ANSI control chars:
< 0x00000: ff fd 18 ff fd 1f ff fb 01 1b 5b 32 4a 1b 5b 3f ..........[2J.[?
< 0x00010: 37 6c 1b 5b 33 3b 32 33 72 1b 5b 3f 36 6c 1b 5b 7l.[3;23r.[?6l.[
< 0x00020: 31 3b 31 48 1b 5b 3f 32 35 6c 1b 5b 31 3b 31 48 1;1H.[?25l.[1;1H
< 0x00030: 48 50 20 4a 39 37 32 38 41 20 32 39 32 30 2d 34 HP J9728A 2920-4
< 0x00040: 38 47 20 53 77 69 74 63 68 0d 0d 0a 53 6f 66 74 8G Switch...Soft
< 0x00050: 77 61 72 65 20 72 65 76 69 73 69 6f 6e 20 57 42 ware revision WB
< 0x00060: 2e 31 35 2e 31 32 2e 30 30 31 35 0d 0d 0a 0d 0d .15.12.0015.....
< 0x00070: 0a 43 6f 70 79 72 69 67 68 74 20 28 43 29 20 31 .Copyright (C) 1
< 0x00080: 39 39 31 2d 32 30 31 34 20 48 65 77 6c 65 74 74 991-2014 Hewlett
< 0x00090: 2d 50 61 63 6b 61 72 64 20 44 65 76 65 6c 6f 70 -Packard Develop
< 0x000a0: 6d 65 6e 74 20 43 6f 6d 70 61 6e 79 2c 20 4c 2e ment Company, L.
< 0x000b0: 50 2e 0d 0a 0d 0a 20 20 20 20 20 20 20 20 20 20 P.....
Unfortunately, this screws up waitfor because if I try to waitfor(/^password:/i) it will return a string with those control chars in it, or wait forever since the regex is never matched.
Is there any way to have Net::Telnet automatically remove those control characters? Is there any way to have waitfor only care about ASCII printable characters?

How do I connect to a websocket manually, with netcat/socat/telnet?

I am trying to connect to the reference websocket echo server "manually", in order to learn how the protocol works (I am using socat for that). However, the server invariably closes the connection without providing an answer. Any idea why?
Here is what I do:
socat - TCP:echo.websocket.org:80
Then, I paste the following text in the terminal:
GET /?encoding=text HTTP/1.1
Origin: http://www.websocket.org
Connection: Upgrade
Host: echo.websocket.org
Sec-WebSocket-Key: P7Kp2hTLNRPFMGLxPV47eQ==
Upgrade: websocket
Sec-WebSocket-Version: 13
I sniffed the parameters of the connection with the developer tools, in firefox, on the same machine, where this works flawlessly: therefore, I would assume they are correct. However after that, the server closes the connection immediately, without providing an answer. Why? How can I implement the protocol "manually"?
I would like type test in my terminal and get the server to reply with what I typed (It works in a web browser).
I think you want to modify the socket stream to translate \n (line feed) to CRLF (Carriage return & line feed). Doing info socat produces detailed information which includes this modifier:
crnl Converts the default line termination character NL ('\n', 0x0a)
to/from CRNL ("\r\n", 0x0d0a) when writing/reading on this chan-
nel (example). Note: socat simply strips all CR characters.
So I think you should be able to do this:
socat - TCP:echo.websocket.org:80,crnl
I'd like to add that my WebSocket tool websocat can help in debugging the WebSocket protocol, especially when combined with socat:
$ websocat - ws-c:sh-c:"socat -v -x - tcp:echo.websocket.org:80" --ws-c-uri ws://echo.websocket.org
> 2018/07/03 16:30:06.021658 length=157 from=0 to=156
47 45 54 20 2f 20 48 54 54 50 2f 31 2e 31 0d 0a GET / HTTP/1.1..
48 6f 73 74 3a 20 65 63 68 6f 2e 77 65 62 73 6f Host: echo.webso
63 6b 65 74 2e 6f 72 67 0d 0a cket.org..
43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 55 70 67 72 Connection: Upgr
61 64 65 0d 0a ade..
55 70 67 72 61 64 65 3a 20 77 65 62 73 6f 63 6b Upgrade: websock
65 74 0d 0a et..
53 65 63 2d 57 65 62 53 6f 63 6b 65 74 2d 56 65 Sec-WebSocket-Ve
72 73 69 6f 6e 3a 20 31 33 0d 0a rsion: 13..
53 65 63 2d 57 65 62 53 6f 63 6b 65 74 2d 4b 65 Sec-WebSocket-Ke
79 3a 20 59 76 36 32 44 31 57 6d 7a 79 79 31 65 y: Yv62D1Wmzyy1e
69 6d 62 47 6d 68 69 61 67 3d 3d 0d 0a imbGmhiag==..
0d 0a ..
--
< 2018/07/03 16:30:06.164057 length=201 from=0 to=200
48 54 54 50 2f 31 2e 31 20 31 30 31 20 57 65 62 HTTP/1.1 101 Web
20 53 6f 63 6b 65 74 20 50 72 6f 74 6f 63 6f 6c Socket Protocol
20 48 61 6e 64 73 68 61 6b 65 0d 0a Handshake..
43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 55 70 67 72 Connection: Upgr
61 64 65 0d 0a ade..
44 61 74 65 3a 20 54 75 65 2c 20 30 33 20 4a 75 Date: Tue, 03 Ju
6c 20 32 30 31 38 20 31 33 3a 31 35 3a 30 30 20 l 2018 13:15:00
47 4d 54 0d 0a GMT..
53 65 63 2d 57 65 62 53 6f 63 6b 65 74 2d 41 63 Sec-WebSocket-Ac
63 65 70 74 3a 20 55 56 6a 32 74 35 50 43 7a 62 cept: UVj2t5PCzb
58 49 32 52 4e 51 75 70 2f 71 48 31 63 5a 44 6e XI2RNQup/qH1cZDn
38 3d 0d 0a 8=..
53 65 72 76 65 72 3a 20 4b 61 61 7a 69 6e 67 20 Server: Kaazing
47 61 74 65 77 61 79 0d 0a Gateway..
55 70 67 72 61 64 65 3a 20 77 65 62 73 6f 63 6b Upgrade: websock
65 74 0d 0a et..
0d 0a ..
--
ABCDEF
> 2018/07/03 16:30:12.707919 length=13 from=157 to=169
82 87 40 57 f5 88 01 15 b6 cc 05 11 ff ..#W.........
--
< 2018/07/03 16:30:12.848398 length=9 from=201 to=209
82 07 41 42 43 44 45 46 0a ..ABCDEF.
--
ABCDEF
> 2018/07/03 16:30:14.528333 length=6 from=170 to=175
88 80 18 ec 05 a8 ......
--
< 2018/07/03 16:30:14.671629 length=2 from=210 to=211
88 00 ..
--
In case of failures with manually driven socat -v -x - TCP:echo.websocket.org:80,crnl (mentioned in the other answer), you can compare it with WebSocat-driven socat like in session depicted above.
Reverse (server) example with socat debug dump:
socat -v -x tcp-l:1234,fork,reuseaddr exec:'websocat -t ws-u\:stdio\: mirror\:'
Alternatively, here is a way to connect and read the stream from a wss secure websocket stream from the command line using solely core php.
php -r '$sock=stream_socket_client("tls://echo.websocket.org:443",$e,$n,30,STREAM_CLIENT_CONNECT,stream_context_create(null));if(!$sock){echo"[$n]$e".PHP_EOL;}else{fwrite($sock,"GET / HTTP/1.1\r\nHost: echo.websocket.org\r\nAccept: */*\r\nConnection: Upgrade\r\nUpgrade: websocket\r\nSec-WebSocket-Version: 13\r\nSec-WebSocket-Key: ".rand(0,999)."\r\n\r\n");while(!feof($sock)){var_dump(fgets($sock,2048));}}'
Other similar example, pulling from another wss server: (Do not get rekt)
php -r '$sock=stream_socket_client("tls://stream.binance.com:9443",$e,$n,30,STREAM_CLIENT_CONNECT,stream_context_create(null));if(!$sock){echo"[$n]$e".PHP_EOL;}else{fwrite($sock,"GET /stream?streams=btcusdt#kline_1m HTTP/1.1\r\nHost: stream.binance.com:9443\r\nAccept: */*\r\nConnection: Upgrade\r\nUpgrade: websocket\r\nSec-WebSocket-Version: 13\r\nSec-WebSocket-Key: ".rand(0,999)."\r\n\r\n");while(!feof($sock)){var_dump(explode(",",fgets($sock,512)));}}'

Resources