Fine Uploader 5.11.10 SignatureDoesNotMatch with PHP & AWS S3 - fine-uploader

UPDATED POST AND RETURN INFO
I'm new to FineUploader and I'm running into problems when uploading files to AWS s3. S3 responds with <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message> I've tried many different options and parameters and have currently paired it down to a very simple configuration but still get the same response. I've followed the steps in the documentation for setting up fineuploder and the PHP signature endpoint for signing, gone through countless options/settings, looked at various posts but nothing has quite got me to a solution.
The js is simplified down to:
var s3Uploader = new qq.s3.FineUploader({
debug: true,
element: document.getElementById('fine_uploader'),
request: {
endpoint: 'https://bucketname.s3.amazonaws.com',
accessKey: 'KEY'
},
cors: {
expected: true,
sendCredentials: true
},
signature: {
endpoint: 'https://domain/s3/endpoint',
version: 4
},
uploadSuccess: {
endpoint: 'https://domain/s3/endpointsuccess'
},
iframeSupport: {
localBlankPagePath: 'https://domain/s3/blank'
},
thumbnails: {
placeholders: {
notAvailablePath: "https://domain/not_available-generic.png",
waitingPath: "https://domain/waiting-generic.png"
}
}
});
The endpoint-cors.php file for signing requests is identical to that found on Github FineUploader/php-s3-server/endpoint-cors.php dated Nov 5, 2015.
FineUploder submits the following to my endpoint for signing:
{"expiration":"2016-12-30T16:58:27.028Z","conditions":[{"acl":"private"},{"bucket":"bucketname"},{"Content-Type":"application/pdf"},{"success_action_status":"200"},{"x-amz-algorithm":"AWS4-HMAC-SHA256"},{"key":"2c6ed85e-1d5f-45e3-8458-e864f92a8cf3.pdf"},{"x-amz-credential":"KEY/20161230/us-east-1/s3/aws4_request"},{"x-amz-date":"20161230T165327Z"},{"x-amz-meta-qqfilename":"DefaultDocument.pdf"}]}
My signing endpoint responds with:
{"policy":"eyJleHBpcmF0aW9uIjoiMjAxNi0xMi0zMFQxNjo1ODoyNy4wMjhaIiwiY29uZGl0aW9ucyI6W3siYWNsIjoicHJpdmF0ZSJ9LHsiYnVja2V0IjoidXBsb2FkdG9idWlsZGVycG9wIn0seyJDb250ZW50LVR5cGUiOiJhcHBsaWNhdGlvblwvcGRmIn0seyJzdWNjZXNzX2FjdGlvbl9zdGF0dXMiOiIyMDAifSx7IngtYW16LWFsZ29yaXRobSI6IkFXUzQtSE1BQy1TSEEyNTYifSx7ImtleSI6IjJjNmVkODVlLTFkNWYtNDVlMy04NDU4LWU4NjRmOTJhOGNmMy5wZGYifSx7IngtYW16LWNyZWRlbnRpYWwiOiJBS0lBSjRaNlRWRDVVNTZOUzVRUVwvMjAxNjEyMzBcL3VzLWVhc3QtMVwvczNcL2F3czRfcmVxdWVzdCJ9LHsieC1hbXotZGF0ZSI6IjIwMTYxMjMwVDE2NTMyN1oifSx7IngtYW16LW1ldGEtcXFmaWxlbmFtZSI6IkRlZmF1bHREb2N1bWVudC5wZGYifV19","signature":"01dd1e6ab8be148c9f26b8721a267f41d0b4e1d6ac1da7ca3f5a1cc8ab9888e3"}
AWS S3 POST information for file upload:
Request URL:https://bucketname.s3.amazonaws.com/
Request Method:POST
Status Code:403 Forbidden
Remote Address:ipaddress:443
Response Headers
Access-Control-Allow-Methods:POST, PUT, DELETE
Access-Control-Allow-Origin:*
Access-Control-Expose-Headers:ETag
Access-Control-Max-Age:3000
Connection:close
Content-Type:application/xml
Date:Fri, 30 Dec 2016 16:53:29 GMT
Server:AmazonS3
Transfer-Encoding:chunked
Vary:Origin, Access-Control-Request-Headers, Access-Control-Request-Method
x-amz-id-2:lQOge5d8wTNIwH1g3TE/IuNPXNad29p6PbC8DHPDTPZzWvS9eTK6wE5nd2K6Kt/f+FsH9B0oBRw=
x-amz-request-id:E9A9C61173E13568
Request Headers
Accept:*/*
Accept-Encoding:gzip, deflate, br
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
Content-Length:659510
Content-Type:multipart/form-data; boundary=----WebKitFormBoundaryiz1gbHgrB4vnnG1m
Host:bucketname.s3.amazonaws.com
Origin:https://domain
Referer:https://domain/create/?pt=O1&pid=mFflJn
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36
Request Payload
------WebKitFormBoundaryiz1gbHgrB4vnnG1m
Content-Disposition: form-data; name="key"
2c6ed85e-1d5f-45e3-8458-e864f92a8cf3.pdf
------WebKitFormBoundaryiz1gbHgrB4vnnG1m
Content-Disposition: form-data; name="Content-Type"
application/pdf
------WebKitFormBoundaryiz1gbHgrB4vnnG1m
Content-Disposition: form-data; name="success_action_status"
200
------WebKitFormBoundaryiz1gbHgrB4vnnG1m
Content-Disposition: form-data; name="acl"
private
------WebKitFormBoundaryiz1gbHgrB4vnnG1m
Content-Disposition: form-data; name="x-amz-meta-qqfilename"
DefaultDocument.pdf
------WebKitFormBoundaryiz1gbHgrB4vnnG1m
Content-Disposition: form-data; name="x-amz-algorithm"
AWS4-HMAC-SHA256
------WebKitFormBoundaryiz1gbHgrB4vnnG1m
Content-Disposition: form-data; name="x-amz-credential"
AKIAJ4Z6TVD5U56NS5QQ/20161230/us-east-1/s3/aws4_request
------WebKitFormBoundaryiz1gbHgrB4vnnG1m
Content-Disposition: form-data; name="x-amz-date"
20161230T165327Z
------WebKitFormBoundaryiz1gbHgrB4vnnG1m
Content-Disposition: form-data; name="policy"
eyJleHBpcmF0aW9uIjoiMjAxNi0xMi0zMFQxNjo1ODoyNy4wMjhaIiwiY29uZGl0aW9ucyI6W3siYWNsIjoicHJpdmF0ZSJ9LHsiYnVja2V0IjoidXBsb2FkdG9idWlsZGVycG9wIn0seyJDb250ZW50LVR5cGUiOiJhcHBsaWNhdGlvblwvcGRmIn0seyJzdWNjZXNzX2FjdGlvbl9zdGF0dXMiOiIyMDAifSx7IngtYW16LWFsZ29yaXRobSI6IkFXUzQtSE1BQy1TSEEyNTYifSx7ImtleSI6IjJjNmVkODVlLTFkNWYtNDVlMy04NDU4LWU4NjRmOTJhOGNmMy5wZGYifSx7IngtYW16LWNyZWRlbnRpYWwiOiJBS0lBSjRaNlRWRDVVNTZOUzVRUVwvMjAxNjEyMzBcL3VzLWVhc3QtMVwvczNcL2F3czRfcmVxdWVzdCJ9LHsieC1hbXotZGF0ZSI6IjIwMTYxMjMwVDE2NTMyN1oifSx7IngtYW16LW1ldGEtcXFmaWxlbmFtZSI6IkRlZmF1bHREb2N1bWVudC5wZGYifV19
------WebKitFormBoundaryiz1gbHgrB4vnnG1m
Content-Disposition: form-data; name="x-amz-signature"
01dd1e6ab8be148c9f26b8721a267f41d0b4e1d6ac1da7ca3f5a1cc8ab9888e3
------WebKitFormBoundaryiz1gbHgrB4vnnG1m
Content-Disposition: form-data; name="file"; filename="DefaultDocument.pdf"
Content-Type: application/pdf
------WebKitFormBoundaryiz1gbHgrB4vnnG1m--
AWS S3 POST RESPONSE:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>AKIAJ4Z6TVD5U56NS5QQ</AWSAccessKeyId><StringToSign>eyJleHBpcmF0aW9uIjoiMjAxNi0xMi0zMFQxNjo1ODoyNy4wMjhaIiwiY29uZGl0aW9ucyI6W3siYWNsIjoicHJpdmF0ZSJ9LHsiYnVja2V0IjoidXBsb2FkdG9idWlsZGVycG9wIn0seyJDb250ZW50LVR5cGUiOiJhcHBsaWNhdGlvblwvcGRmIn0seyJzdWNjZXNzX2FjdGlvbl9zdGF0dXMiOiIyMDAifSx7IngtYW16LWFsZ29yaXRobSI6IkFXUzQtSE1BQy1TSEEyNTYifSx7ImtleSI6IjJjNmVkODVlLTFkNWYtNDVlMy04NDU4LWU4NjRmOTJhOGNmMy5wZGYifSx7IngtYW16LWNyZWRlbnRpYWwiOiJBS0lBSjRaNlRWRDVVNTZOUzVRUVwvMjAxNjEyMzBcL3VzLWVhc3QtMVwvczNcL2F3czRfcmVxdWVzdCJ9LHsieC1hbXotZGF0ZSI6IjIwMTYxMjMwVDE2NTMyN1oifSx7IngtYW16LW1ldGEtcXFmaWxlbmFtZSI6IkRlZmF1bHREb2N1bWVudC5wZGYifV19</StringToSign><SignatureProvided>01dd1e6ab8be148c9f26b8721a267f41d0b4e1d6ac1da7ca3f5a1cc8ab9888e3</SignatureProvided><StringToSignBytes>65 79 4a 6c 65 48 42 70 63 6d 46 30 61 57 39 75 49 6a 6f 69 4d 6a 41 78 4e 69 30 78 4d 69 30 7a 4d 46 51 78 4e 6a 6f 31 4f 44 6f 79 4e 79 34 77 4d 6a 68 61 49 69 77 69 59 32 39 75 5a 47 6c 30 61 57 39 75 63 79 49 36 57 33 73 69 59 57 4e 73 49 6a 6f 69 63 48 4a 70 64 6d 46 30 5a 53 4a 39 4c 48 73 69 59 6e 56 6a 61 32 56 30 49 6a 6f 69 64 58 42 73 62 32 46 6b 64 47 39 69 64 57 6c 73 5a 47 56 79 63 47 39 77 49 6e 30 73 65 79 4a 44 62 32 35 30 5a 57 35 30 4c 56 52 35 63 47 55 69 4f 69 4a 68 63 48 42 73 61 57 4e 68 64 47 6c 76 62 6c 77 76 63 47 52 6d 49 6e 30 73 65 79 4a 7a 64 57 4e 6a 5a 58 4e 7a 58 32 46 6a 64 47 6c 76 62 6c 39 7a 64 47 46 30 64 58 4d 69 4f 69 49 79 4d 44 41 69 66 53 78 37 49 6e 67 74 59 57 31 36 4c 57 46 73 5a 32 39 79 61 58 52 6f 62 53 49 36 49 6b 46 58 55 7a 51 74 53 45 31 42 51 79 31 54 53 45 45 79 4e 54 59 69 66 53 78 37 49 6d 74 6c 65 53 49 36 49 6a 4a 6a 4e 6d 56 6b 4f 44 56 6c 4c 54 46 6b 4e 57 59 74 4e 44 56 6c 4d 79 30 34 4e 44 55 34 4c 57 55 34 4e 6a 52 6d 4f 54 4a 68 4f 47 4e 6d 4d 79 35 77 5a 47 59 69 66 53 78 37 49 6e 67 74 59 57 31 36 4c 57 4e 79 5a 57 52 6c 62 6e 52 70 59 57 77 69 4f 69 4a 42 53 30 6c 42 53 6a 52 61 4e 6c 52 57 52 44 56 56 4e 54 5a 4f 55 7a 56 52 55 56 77 76 4d 6a 41 78 4e 6a 45 79 4d 7a 42 63 4c 33 56 7a 4c 57 56 68 63 33 51 74 4d 56 77 76 63 7a 4e 63 4c 32 46 33 63 7a 52 66 63 6d 56 78 64 57 56 7a 64 43 4a 39 4c 48 73 69 65 43 31 68 62 58 6f 74 5a 47 46 30 5a 53 49 36 49 6a 49 77 4d 54 59 78 4d 6a 4d 77 56 44 45 32 4e 54 4d 79 4e 31 6f 69 66 53 78 37 49 6e 67 74 59 57 31 36 4c 57 31 6c 64 47 45 74 63 58 46 6d 61 57 78 6c 62 6d 46 74 5a 53 49 36 49 6b 52 6c 5a 6d 46 31 62 48 52 45 62 32 4e 31 62 57 56 75 64 43 35 77 5a 47 59 69 66 56 31 39</StringToSignBytes><RequestId>E9A9C61173E13568</RequestId><HostId>lQOge5d8wTNIwH1g3TE/IuNPXNad29p6PbC8DHPDTPZzWvS9eTK6wE5nd2K6Kt/f+FsH9B0oBRw=</HostId></Error>
Comparison of the endpoint signature and the AWS S3 POST response appear the same signature and policy are being submitted.
Response Snipets:
endpoint response: {....,"signature":"01dd1e6ab8be148c9f26b8721a267f41d0b4e1d6ac1da7ca3f5a1cc8ab9888e3"}
AWS S3 response: <SignatureProvided>01dd1e6ab8be148c9f26b8721a267f41d0b4e1d6ac1da7ca3f5a1cc8ab9888e3</SignatureProvided>
endpoint policy: eyJleHBpcmF0aW9uIjoiMjAxNi0xMi0zMFQxNjo1ODoyNy4wMjhaIiwiY29uZGl0aW9ucyI6W3siYWNsIjoicHJpdmF0ZSJ9LHsiYnVja2V0IjoidXBsb2FkdG9idWlsZGVycG9wIn0seyJDb250ZW50LVR5cGUiOiJhcHBsaWNhdGlvblwvcGRmIn0seyJzdWNjZXNzX2FjdGlvbl9zdGF0dXMiOiIyMDAifSx7IngtYW16LWFsZ29yaXRobSI6IkFXUzQtSE1BQy1TSEEyNTYifSx7ImtleSI6IjJjNmVkODVlLTFkNWYtNDVlMy04NDU4LWU4NjRmOTJhOGNmMy5wZGYifSx7IngtYW16LWNyZWRlbnRpYWwiOiJBS0lBSjRaNlRWRDVVNTZOUzVRUVwvMjAxNjEyMzBcL3VzLWVhc3QtMVwvczNcL2F3czRfcmVxdWVzdCJ9LHsieC1hbXotZGF0ZSI6IjIwMTYxMjMwVDE2NTMyN1oifSx7IngtYW16LW1ldGEtcXFmaWxlbmFtZSI6IkRlZmF1bHREb2N1bWVudC5wZGYifV19
S3 Response: <StringToSign>eyJleHBpcmF0aW9uIjoiMjAxNi0xMi0zMFQxNjo1ODoyNy4wMjhaIiwiY29uZGl0aW9ucyI6W3siYWNsIjoicHJpdmF0ZSJ9LHsiYnVja2V0IjoidXBsb2FkdG9idWlsZGVycG9wIn0seyJDb250ZW50LVR5cGUiOiJhcHBsaWNhdGlvblwvcGRmIn0seyJzdWNjZXNzX2FjdGlvbl9zdGF0dXMiOiIyMDAifSx7IngtYW16LWFsZ29yaXRobSI6IkFXUzQtSE1BQy1TSEEyNTYifSx7ImtleSI6IjJjNmVkODVlLTFkNWYtNDVlMy04NDU4LWU4NjRmOTJhOGNmMy5wZGYifSx7IngtYW16LWNyZWRlbnRpYWwiOiJBS0lBSjRaNlRWRDVVNTZOUzVRUVwvMjAxNjEyMzBcL3VzLWVhc3QtMVwvczNcL2F3czRfcmVxdWVzdCJ9LHsieC1hbXotZGF0ZSI6IjIwMTYxMjMwVDE2NTMyN1oifSx7IngtYW16LW1ldGEtcXFmaWxlbmFtZSI6IkRlZmF1bHREb2N1bWVudC5wZGYifV19</StringToSign>
Thanks for any help you provide.

Related

Signature Mismatch error when using S3 Signed URL

I don't know what happened in the last 24 hours that caused this thing to break, but it had been working perfectly for 3 years, non stop without issues. Trust me, I haven't changed any code. As a matter of fact, I haven't pushed code for 2 weeks.
Basically, I am using Laravel 6 and Laravel Vapor to upload files to S3 using the Signed URLs mechanism that the framework provides. Ever since this morning, I am seeing the following issue. The AWS PHP SDK is 3.148.3, by the way.
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
<AWSAccessKeyId>[REDACTED]</AWSAccessKeyId>
<StringToSign>AWS4-HMAC-SHA256 20220712T203731Z 20220712/eu-west-2/s3/aws4_request 961d5d32676be187af1a7ccf8a92b8c29e2d851bdd51c46836aedce7cc9e089a</StringToSign>
<SignatureProvided>101e80ad27bfe4ed12a34b19fff4c17f87a3f639b92d62c8e7a79b13c876b1ca</SignatureProvided>
<StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 32 30 37 31 32 54 32 30 33 37 33 31 5a 0a 32 30 32 32 30 37 31 32 2f 65 75 2d 77 65 73 74 2d 32 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 39 36 31 64 35 64 33 32 36 37 36 62 65 31 38 37 61 66 31 61 37 63 63 66 38 61 39 32 62 38 63 32 39 65 32 64 38 35 31 62 64 64 35 31 63 34 36 38 33 36 61 65 64 63 65 37 63 63 39 65 30 38 39 61</StringToSignBytes>
<CanonicalRequest>GET /tmp/0344aa31-c756-49c3-8ce1-932ec23c895d X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=[REDACTED]%2F20220712%2Feu-west-2%2Fs3%2Faws4_request&X-Amz-Date=20220712T203731Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host%3Bx-amz-acl&x-amz-acl=public-read host:[REDACTED].s3.eu-west-2.amazonaws.com x-amz-acl:public-read host;x-amz-acl UNSIGNED-PAYLOAD</CanonicalRequest>
<CanonicalRequestBytes>47 45 54 0a 2f 74 6d 70 2f 30 33 34 34 61 61 33 31 2d 63 37 35 36 2d 34 39 63 33 2d 38 63 65 31 2d 39 33 32 65 63 32 33 63 38 39 35 64 0a 58 2d 41 6d 7a 2d 41 6c 67 6f 72 69 74 68 6d 3d 41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 26 58 2d 41 6d 7a 2d 43 6f 6e 74 65 6e 74 2d 53 68 61 32 35 36 3d 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44 26 58 2d 41 6d 7a 2d 43 72 65 64 65 6e 74 69 61 6c 3d 41 4b 49 41 34 58 4b 4a 50 55 57 34 55 43 42 4a 49 32 57 57 25 32 46 32 30 32 32 30 37 31 32 25 32 46 65 75 2d 77 65 73 74 2d 32 25 32 46 73 33 25 32 46 61 77 73 34 5f 72 65 71 75 65 73 74 26 58 2d 41 6d 7a 2d 44 61 74 65 3d 32 30 32 32 30 37 31 32 54 32 30 33 37 33 31 5a 26 58 2d 41 6d 7a 2d 45 78 70 69 72 65 73 3d 33 30 30 26 58 2d 41 6d 7a 2d 53 69 67 6e 65 64 48 65 61 64 65 72 73 3d 68 6f 73 74 25 33 42 78 2d 61 6d 7a 2d 61 63 6c 26 78 2d 61 6d 7a 2d 61 63 6c 3d 70 75 62 6c 69 63 2d 72 65 61 64 0a 68 6f 73 74 3a 73 68 61 64 6f 77 66 6f 75 6e 64 72 2d 75 70 6c 6f 61 64 73 2d 6c 6f 63 61 6c 2e 73 33 2e 65 75 2d 77 65 73 74 2d 32 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 78 2d 61 6d 7a 2d 61 63 6c 3a 70 75 62 6c 69 63 2d 72 65 61 64 0a 0a 68 6f 73 74 3b 78 2d 61 6d 7a 2d 61 63 6c 0a 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44</CanonicalRequestBytes>
<RequestId>S6E8RFNVQ3WGAAHB</RequestId>
<HostId>yA998rWcqzfVIdD9HyrIJRJtbI/ge5opCEHEgpPoBBAuwRjgABub590OnA66JVHwyxHdo/a2uB6+ii/IxwZIGw==</HostId>
</Error>
Vue.js
#change=upload
convertFileNameToKebabCase(fileNameRaw) {
let fileNameActual = _.kebabCase(fileNameRaw.split(".").shift());
let fileExtension = fileNameRaw.split(".").pop();
return `${fileNameActual}.${fileExtension}`;
},
async store(file, options = {}, fileAcl) {
const response = await Vue.axios.post(
"/vapor/signed-storage-url", {
bucket: options.bucket || "",
content_type: options.contentType || file.type,
expires: options.expires || "",
visibility: fileAcl || "",
ignore_web_middleware: true
}, {
baseURL: options.baseURL || null,
headers: options.headers || {}
}
);
let headers = response.data.headers;
if ("Host" in headers) {
delete headers.Host;
}
if (typeof options.progress === "undefined") {
options.progress = () => {};
}
// Reset axios instance as Amazon
// does not like our Auth header
const newInstance = axios.create();
await newInstance.put(response.data.url, file, {
headers: headers,
onUploadProgress: progressEvent => {
options.progress(
progressEvent.loaded / progressEvent.total
);
}
});
response.data.extension = file.name.split(".").pop();
return response.data;
},
async upload(event) {
const {
valid
} = await this.$refs.provider.validate(event);
if (valid) {
this.reset();
let fileData = this.$refs.file.files[0];
this.fileName = fileData.name;
this.url = false;
try {
const response = await this.store(
this.$refs.file.files[0], {
progress: progress => {
this.uploadProgress = Math.round(progress * 100);
}
},
"public-read"
);
Vue.axios
.post("/file/commit", {
uuid: response.uuid,
key: response.key,
bucket: response.bucket,
name: this.convertFileNameToKebabCase(this.$refs.file.files[0].name),
content_type: this.$refs.file.files[0].type
})
.then(success => {
this.reset();
this.url = success.data.data.file;
this.uploadCompleted = true;
})
.catch(error => {
this.reset();
this.uploadFailed = true;
});
} catch (error) {
this.reset();
this.uploadFailed = true;
}
}
},
Three things to check:
Region mismatch, the bucket is in region X but CLI/SDK goes to Region Y.
A corruption or unusual setting in your AWS CLI profile, reset the
profile, check if access key and secret are still valid.
CORS configuration not containing <AllowedHeader>*</AllowedHeader>
Otherwise could you share some more details on the code that causes the error?
I found the solution. I had to remove the following in the headers: x-amz-acl. I do not know why this was the problem but it seems to have worked. Hope if anyone comes across this in the future, try this solution.
if ("Host" in headers) {
delete headers.Host;
delete headers["x-amz-acl"];
}

How to read a hexadecimal file and convert the content to byte slice in golang?

The hexadecimal file is from the charles proxy's hex format, and it may contains invisible characters, the example format content is:
00000000 7b 22 73 75 70 70 6f 72 74 73 5f 69 6d 70 6c 69 {"supports_impli
00000010 63 69 74 5f 73 64 6b 5f 6c 6f 67 67 69 6e 67 22 cit_sdk_logging"
00000020 3a 74 72 75 65 2c 22 67 64 70 76 34 5f 6e 75 78 :true,"gdpv4_nux
00000030 5f 65 6e 61 62 6c 65 64 22 3a 66 61 6c 73 65 2c _enabled":false,
00000040 22 61 6e 64 72 6f 69 64 5f 73 64 6b 5f 65 72 72 "android_sdk_err
00000050 6f 72 5f 63 61 74 65 67 6f 72 69 65 73 22 3a 5b or_categories":[
00000060 7b 22 6e 61 6d 65 22 3a 22 6c 6f 67 69 6e 5f 72 {"name":"login_r
00000070 65 63 6f 76 65 72 61 62 6c 65 22 2c 22 69 74 65 ecoverable","ite
00000080 6d 73 22 3a 5b 7b 22 63 6f 64 65 22 3a 31 30 32 ms":[{"code":102
00000090 7d 2c 7b 22 63 6f 64 65 22 3a 31 39 30 7d 5d 2c },{"code":190}],
000000a0 22 72 65 63 6f 76 65 72 79 5f 6d 65 73 73 61 67 "recovery_messag
000000b0 65 22 3a 22 5c 75 38 62 66 37 5c 75 39 31 63 64 e":"\u8bf7\u91cd
000000c0 5c 75 36 35 62 30 5c 75 37 36 37 62 5c 75 35 66 \u65b0\u767b\u5f
000000d0 35 35 5c 75 35 65 39 34 5c 75 37 35 32 38 5c 75 55\u5e94\u7528\u
000000e0 37 61 30 62 5c 75 35 65 38 66 5c 75 66 66 30 63 7a0b\u5e8f\uff0c
000000f0 5c 75 35 31 38 64 5c 75 36 62 32 31 5c 75 38 66 \u518d\u6b21\u8f
00000100 64 65 5c 75 36 33 61 35 20 46 61 63 65 62 6f 6f de\u63a5 Faceboo
00000110 6b 20 5c 75 35 65 31 30 5c 75 36 32 33 37 5c 75 k \u5e10\u6237\u
00000120 33 30 30 32 22 7d 5d 2c 22 61 70 70 5f 65 76 65 3002"}],"app_eve
00000130 6e 74 73 5f 73 65 73 73 69 6f 6e 5f 74 69 6d 65 nts_session_time
00000140 6f 75 74 22 3a 36 30 2c 22 61 70 70 5f 65 76 65 out":60,"app_eve
00000150 6e 74 73 5f 66 65 61 74 75 72 65 5f 62 69 74 6d nts_feature_bitm
00000160 61 73 6b 22 3a 36 35 35 35 39 2c 22 73 65 61 6d ask":65559,"seam
00000170 6c 65 73 73 5f 6c 6f 67 69 6e 22 3a 31 2c 22 73 less_login":1,"s
00000180 6d 61 72 74 5f 6c 6f 67 69 6e 5f 62 6f 6f 6b 6d mart_login_bookm
00000190 61 72 6b 5f 69 63 6f 6e 5f 75 72 6c 22 3a 22 68 ark_icon_url":"h
000001a0 74 74 70 73 3a 5c 2f 5c 2f 73 74 61 74 69 63 2e ttps:\/\/static.
000001b0 78 78 2e 66 62 63 64 6e 2e 6e 65 74 5c 2f 72 73 xx.fbcdn.net\/rs
000001c0 72 63 2e 70 68 70 5c 2f 76 33 5c 2f 79 73 5c 2f rc.php\/v3\/ys\/
000001d0 72 5c 2f 43 36 5a 75 74 59 44 53 61 61 56 2e 70 r\/C6ZutYDSaaV.p
000001e0 6e 67 22 2c 22 73 6d 61 72 74 5f 6c 6f 67 69 6e ng","smart_login
000001f0 5f 6d 65 6e 75 5f 69 63 6f 6e 5f 75 72 6c 22 3a _menu_icon_url":
00000200 22 68 74 74 70 73 3a 5c 2f 5c 2f 73 74 61 74 69 "https:\/\/stati
00000210 63 2e 78 78 2e 66 62 63 64 6e 2e 6e 65 74 5c 2f c.xx.fbcdn.net\/
00000220 72 73 72 63 2e 70 68 70 5c 2f 76 33 5c 2f 79 73 rsrc.php\/v3\/ys
00000230 5c 2f 72 5c 2f 30 69 61 72 70 6e 77 64 6d 45 78 \/r\/0iarpnwdmEx
00000240 2e 70 6e 67 22 2c 22 72 65 73 74 72 69 63 74 69 .png","restricti
00000250 76 65 5f 64 61 74 61 5f 66 69 6c 74 65 72 5f 70 ve_data_filter_p
00000260 61 72 61 6d 73 22 3a 22 7b 7d 22 2c 22 61 61 6d arams":"{}","aam
00000270 5f 72 75 6c 65 73 22 3a 22 7b 7d 22 2c 22 73 75 _rules":"{}","su
00000280 67 67 65 73 74 65 64 5f 65 76 65 6e 74 73 5f 73 ggested_events_s
00000290 65 74 74 69 6e 67 22 3a 22 7b 5c 22 70 72 6f 64 etting":"{\"prod
000002a0 75 63 74 69 6f 6e 5f 65 76 65 6e 74 73 5c 22 3a uction_events\":
000002b0 5b 5d 2c 5c 22 65 6c 69 67 69 62 6c 65 5f 66 6f [],\"eligible_fo
000002c0 72 5f 70 72 65 64 69 63 74 69 6f 6e 5f 65 76 65 r_prediction_eve
000002d0 6e 74 73 5c 22 3a 5b 5c 22 66 62 5f 6d 6f 62 69 nts\":[\"fb_mobi
000002e0 6c 65 5f 61 64 64 5f 74 6f 5f 63 61 72 74 5c 22 le_add_to_cart\"
000002f0 2c 5c 22 66 62 5f 6d 6f 62 69 6c 65 5f 70 75 72 ,\"fb_mobile_pur
00000300 63 68 61 73 65 5c 22 2c 5c 22 66 62 5f 6d 6f 62 chase\",\"fb_mob
00000310 69 6c 65 5f 63 6f 6d 70 6c 65 74 65 5f 72 65 67 ile_complete_reg
00000320 69 73 74 72 61 74 69 6f 6e 5c 22 2c 5c 22 66 62 istration\",\"fb
00000330 5f 6d 6f 62 69 6c 65 5f 69 6e 69 74 69 61 74 65 _mobile_initiate
00000340 64 5f 63 68 65 63 6b 6f 75 74 5c 22 5d 7d 22 2c d_checkout\"]}",
00000350 22 69 64 22 3a 22 31 36 33 35 34 33 35 31 34 39 "id":"1635435149
00000360 30 39 30 34 35 22 7d 09045"}
Read the file and convert the hexadecimal content to golang slice []byte, is there any convenient method? how to split the lines in the file?
I have solved it by using fixed length of the hexadecimal content. the offset length of each line is 8, so the hexadecimal content begins at offset 8, and the max length of the hexadecimal is 57. The code is as follows:
const (
beginOffset = 8 // the offset of the hexadecimal content begins
endLength = 57 // the length of the hexadecimal content ends
)
func ReadHexData(src []byte) ([]byte, error) {
lines := strings.Split(string(src), "\n")
var data string
for _, line := range lines {
for i := beginOffset; i < endLength; i++ {
if line[i] != ' ' {
data = data + string(line[i])
}
}
}
return hex.DecodeString(data)
}

How to send delete request with custom header using Spring's WebClient?

I want to send an http delete request with custom header.
I tried to do this with Spring's WebClient, with following code:
#Test
public void validateDeleteCouldCarryHeader() {
WebClient.create("https://jira.spring.io/")
.delete()
.header("X-FOO", "BAR")
.retrieve()
.bodyToMono(Map.class)
.block();
}
I am expecting something like:
DELETE / HTTP/1.1
X-FOO: BAR
...
content-length: 0
But, the actual request made is:
DELETE / HTTP/1.1
user-agent: ReactorNetty/0.7.0.RELEASE
host: jira.spring.io
accept: */*
accept-encoding: gzip
content-length: 0
Did I miss some thing to include the custom header X-FOO ?
Update 1: I upgraded reactor-netty to 0.7.3.RELEASE,
And the missing header is still missing:
DELETE / HTTP/1.1
user-agent: ReactorNetty/0.7.3.RELEASE
host: jira.spring.io
accept: */*
accept-encoding: gzip
content-length: 0
2018-12-28 22:32:32.813 DEBUG 12064 --- [ctor-http-nio-4] r.ipc.netty.http.client.HttpClient : [id: 0x2c6a9cea, L:/172.17.1.131:54532 - R:jira.spring.io/35.199.60.33:443] WRITE: 138B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 44 45 4c 45 54 45 20 2f 20 48 54 54 50 2f 31 2e |DELETE / HTTP/1.|
|00000010| 31 0d 0a 75 73 65 72 2d 61 67 65 6e 74 3a 20 52 |1..user-agent: R|
|00000020| 65 61 63 74 6f 72 4e 65 74 74 79 2f 30 2e 37 2e |eactorNetty/0.7.|
|00000030| 33 2e 52 45 4c 45 41 53 45 0d 0a 68 6f 73 74 3a |3.RELEASE..host:|
|00000040| 20 6a 69 72 61 2e 73 70 72 69 6e 67 2e 69 6f 0d | jira.spring.io.|
|00000050| 0a 61 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 61 63 |.accept: */*..ac|
|00000060| 63 65 70 74 2d 65 6e 63 6f 64 69 6e 67 3a 20 67 |cept-encoding: g|
|00000070| 7a 69 70 0d 0a 63 6f 6e 74 65 6e 74 2d 6c 65 6e |zip..content-len|
|00000080| 67 74 68 3a 20 30 0d 0a 0d 0a |gth: 0.... |
+--------+-------------------------------------------------+----------------+
Update 2: I found there's a reactor's BOM (Bismuth-RELEASE) imported in my pom, after removed that BOM, Spring Boot 2.0.5.RELEASE included the custom header I want.
DELETE / HTTP/1.1
user-agent: ReactorNetty/0.7.9.RELEASE
host: jira.spring.io
accept: */*
accept-encoding: gzip
X-FOO: BAR
Reactor Netty 0.7.0.RELEASE is quite old now, and you should upgrade to a newer version.
I've tried to reproduce this and couldn't.
Setting the log level of reactor.ipc.netty.http.client.HttpClient to "DEBUG" shows this:
[reactor-http-nio-4] DEBUG reactor.ipc.netty.http.client.HttpClient - [id: 0x69202b84, L:/192.168.0.28:60880 - R:jira.spring.io/35.199.60.33:443] WRITE: 150B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 44 45 4c 45 54 45 20 2f 20 48 54 54 50 2f 31 2e |DELETE / HTTP/1.|
|00000010| 31 0d 0a 75 73 65 72 2d 61 67 65 6e 74 3a 20 52 |1..user-agent: R|
|00000020| 65 61 63 74 6f 72 4e 65 74 74 79 2f 30 2e 37 2e |eactorNetty/0.7.|
|00000030| 30 2e 52 45 4c 45 41 53 45 0d 0a 68 6f 73 74 3a |0.RELEASE..host:|
|00000040| 20 6a 69 72 61 2e 73 70 72 69 6e 67 2e 69 6f 0d | jira.spring.io.|
|00000050| 0a 61 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 61 63 |.accept: */*..ac|
|00000060| 63 65 70 74 2d 65 6e 63 6f 64 69 6e 67 3a 20 67 |cept-encoding: g|
|00000070| 7a 69 70 0d 0a 58 2d 46 4f 4f 3a 20 42 41 52 0d |zip..X-FOO: BAR.|
|00000080| 0a 63 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 3a |.content-length:|
|00000090| 20 30 0d 0a 0d 0a | 0.... |
+--------+-------------------------------------------------+----------------+
So the client is indeed sending those headers. Maybe there's something wrong with the way you're capturing the outgoing request information?

Chunk size appears on Browser page

I'm implementing a small web server into a wifi micro. To aid in development and test, I have ported it to Windows console program.
I use chunked transfer processing. The following is what shows up on the browser:
0059
Hello World
0
The 59 is the hex size of the chunk and the 0 is the chunked terminating size
This is the data captured via wireshark:
This is the first message I send which are the headers
0000 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d HTTP/1.1 200 OK.
0010 0a 53 65 72 76 65 72 3a 20 54 72 61 6e 73 66 65 .Server: Transfe
0020 72 2d 45 6e 63 6f 64 69 6e 67 3a 20 63 68 75 6e r-Encoding: chun
0030 6b 65 64 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 ked..Content-Typ
0040 65 3a 20 74 65 78 74 2f 68 74 6d 6c 0d 0a 43 61 e: text/html..Ca
0050 63 68 65 2d 43 6f 6e 74 72 6f 6c 3a 20 6d 61 78 che-Control: max
0060 2d 61 67 65 3d 33 36 30 30 2c 20 6d 75 73 74 2d -age=3600, must-
0070 72 65 76 61 6c 69 64 61 74 65 0d 0a 0d 0a revalidate....
The next block is the chunked data
0000 30 30 35 39 0d 0a 3c 68 74 6d 6c 3e 0a 3c 68 65 0059..<html>.<he
0010 61 64 3e 3c 74 69 74 6c 65 3e 57 65 62 20 53 65 ad><title>Web Se
0020 72 76 65 72 3c 2f 74 69 74 6c 65 3e 0a 3c 2f 68 rver</title>.</h
0030 65 61 64 3e 0a 3c 62 6f 64 79 3e 0a 3c 68 31 3e ead>.<body>.<h1>
0040 48 65 6c 6c 6f 20 57 6f 72 6c 64 3c 2f 68 31 3e Hello World</h1>
0050 0a 3c 2f 62 6f 64 79 3e 3c 2f 68 74 6d 6c 3e 0d .</body></html>.
0060 0a 30 0d 0a 0d 0a .0....
The chunked values are being displayed on both Chrome and IE.
Can anyone see an issue with my data that would cause the issue.
Thanks
Solved:
I mistakenly remove the server name so now the browser is taking the transfer encoding as the server name and does not understand the chunked message size -- it thinks its just data to display.

How do I connect to a websocket manually, with netcat/socat/telnet?

I am trying to connect to the reference websocket echo server "manually", in order to learn how the protocol works (I am using socat for that). However, the server invariably closes the connection without providing an answer. Any idea why?
Here is what I do:
socat - TCP:echo.websocket.org:80
Then, I paste the following text in the terminal:
GET /?encoding=text HTTP/1.1
Origin: http://www.websocket.org
Connection: Upgrade
Host: echo.websocket.org
Sec-WebSocket-Key: P7Kp2hTLNRPFMGLxPV47eQ==
Upgrade: websocket
Sec-WebSocket-Version: 13
I sniffed the parameters of the connection with the developer tools, in firefox, on the same machine, where this works flawlessly: therefore, I would assume they are correct. However after that, the server closes the connection immediately, without providing an answer. Why? How can I implement the protocol "manually"?
I would like type test in my terminal and get the server to reply with what I typed (It works in a web browser).
I think you want to modify the socket stream to translate \n (line feed) to CRLF (Carriage return & line feed). Doing info socat produces detailed information which includes this modifier:
crnl Converts the default line termination character NL ('\n', 0x0a)
to/from CRNL ("\r\n", 0x0d0a) when writing/reading on this chan-
nel (example). Note: socat simply strips all CR characters.
So I think you should be able to do this:
socat - TCP:echo.websocket.org:80,crnl
I'd like to add that my WebSocket tool websocat can help in debugging the WebSocket protocol, especially when combined with socat:
$ websocat - ws-c:sh-c:"socat -v -x - tcp:echo.websocket.org:80" --ws-c-uri ws://echo.websocket.org
> 2018/07/03 16:30:06.021658 length=157 from=0 to=156
47 45 54 20 2f 20 48 54 54 50 2f 31 2e 31 0d 0a GET / HTTP/1.1..
48 6f 73 74 3a 20 65 63 68 6f 2e 77 65 62 73 6f Host: echo.webso
63 6b 65 74 2e 6f 72 67 0d 0a cket.org..
43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 55 70 67 72 Connection: Upgr
61 64 65 0d 0a ade..
55 70 67 72 61 64 65 3a 20 77 65 62 73 6f 63 6b Upgrade: websock
65 74 0d 0a et..
53 65 63 2d 57 65 62 53 6f 63 6b 65 74 2d 56 65 Sec-WebSocket-Ve
72 73 69 6f 6e 3a 20 31 33 0d 0a rsion: 13..
53 65 63 2d 57 65 62 53 6f 63 6b 65 74 2d 4b 65 Sec-WebSocket-Ke
79 3a 20 59 76 36 32 44 31 57 6d 7a 79 79 31 65 y: Yv62D1Wmzyy1e
69 6d 62 47 6d 68 69 61 67 3d 3d 0d 0a imbGmhiag==..
0d 0a ..
--
< 2018/07/03 16:30:06.164057 length=201 from=0 to=200
48 54 54 50 2f 31 2e 31 20 31 30 31 20 57 65 62 HTTP/1.1 101 Web
20 53 6f 63 6b 65 74 20 50 72 6f 74 6f 63 6f 6c Socket Protocol
20 48 61 6e 64 73 68 61 6b 65 0d 0a Handshake..
43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 55 70 67 72 Connection: Upgr
61 64 65 0d 0a ade..
44 61 74 65 3a 20 54 75 65 2c 20 30 33 20 4a 75 Date: Tue, 03 Ju
6c 20 32 30 31 38 20 31 33 3a 31 35 3a 30 30 20 l 2018 13:15:00
47 4d 54 0d 0a GMT..
53 65 63 2d 57 65 62 53 6f 63 6b 65 74 2d 41 63 Sec-WebSocket-Ac
63 65 70 74 3a 20 55 56 6a 32 74 35 50 43 7a 62 cept: UVj2t5PCzb
58 49 32 52 4e 51 75 70 2f 71 48 31 63 5a 44 6e XI2RNQup/qH1cZDn
38 3d 0d 0a 8=..
53 65 72 76 65 72 3a 20 4b 61 61 7a 69 6e 67 20 Server: Kaazing
47 61 74 65 77 61 79 0d 0a Gateway..
55 70 67 72 61 64 65 3a 20 77 65 62 73 6f 63 6b Upgrade: websock
65 74 0d 0a et..
0d 0a ..
--
ABCDEF
> 2018/07/03 16:30:12.707919 length=13 from=157 to=169
82 87 40 57 f5 88 01 15 b6 cc 05 11 ff ..#W.........
--
< 2018/07/03 16:30:12.848398 length=9 from=201 to=209
82 07 41 42 43 44 45 46 0a ..ABCDEF.
--
ABCDEF
> 2018/07/03 16:30:14.528333 length=6 from=170 to=175
88 80 18 ec 05 a8 ......
--
< 2018/07/03 16:30:14.671629 length=2 from=210 to=211
88 00 ..
--
In case of failures with manually driven socat -v -x - TCP:echo.websocket.org:80,crnl (mentioned in the other answer), you can compare it with WebSocat-driven socat like in session depicted above.
Reverse (server) example with socat debug dump:
socat -v -x tcp-l:1234,fork,reuseaddr exec:'websocat -t ws-u\:stdio\: mirror\:'
Alternatively, here is a way to connect and read the stream from a wss secure websocket stream from the command line using solely core php.
php -r '$sock=stream_socket_client("tls://echo.websocket.org:443",$e,$n,30,STREAM_CLIENT_CONNECT,stream_context_create(null));if(!$sock){echo"[$n]$e".PHP_EOL;}else{fwrite($sock,"GET / HTTP/1.1\r\nHost: echo.websocket.org\r\nAccept: */*\r\nConnection: Upgrade\r\nUpgrade: websocket\r\nSec-WebSocket-Version: 13\r\nSec-WebSocket-Key: ".rand(0,999)."\r\n\r\n");while(!feof($sock)){var_dump(fgets($sock,2048));}}'
Other similar example, pulling from another wss server: (Do not get rekt)
php -r '$sock=stream_socket_client("tls://stream.binance.com:9443",$e,$n,30,STREAM_CLIENT_CONNECT,stream_context_create(null));if(!$sock){echo"[$n]$e".PHP_EOL;}else{fwrite($sock,"GET /stream?streams=btcusdt#kline_1m HTTP/1.1\r\nHost: stream.binance.com:9443\r\nAccept: */*\r\nConnection: Upgrade\r\nUpgrade: websocket\r\nSec-WebSocket-Version: 13\r\nSec-WebSocket-Key: ".rand(0,999)."\r\n\r\n");while(!feof($sock)){var_dump(explode(",",fgets($sock,512)));}}'

Resources