How to generate an aws4 signed request in bash? - bash

I'm trying to connect to the Scaleway S3 api in bash (it's a subset of aws s3 api). The signature model is aws4_request, and is documented. In the code below I'm uploading a file. I've reviewed every line and although I'm no bash guru, I'm pretty confident I have all the right parts to compute a correct signature, but I still get 403 error as a response. The request itself seems fine, as far as I can see in a wireshark capture. Can anyone find any issue here?
#!/bin/bash
set -e
echo "creds $SCW_ACCESS_KEY/$SCW_SECRET_KEY"
BUCKET="vni-backups"
REGION="fr-par"
HOST="$BUCKET.s3.$REGION.scw.cloud"
filepath="../import.yml"
file_sha256=$(sha256sum -b $filepath | head -c64)
fulldate=$(date +"%Y%m%dT%H%M%SZ" --utc)
shortdate=$(date +"%Y%m%d" --utc)
# Assemble canonical url
canonicalRequest="PUT
/import.yml
content-type:text/plain
host:$HOST
x-amz-content-sha256:$file_sha256
x-amz-date:$fulldate
content-type;host;x-amz-content-sha256;x-amz-date
$file_sha256"
canonReqSha=$(echo -n "$canonicalRequest" | openssl dgst -sha256 | awk '{print $2}')
echo $canonicalRequest
echo "requestHash: $canonReqSha"
echo "
---------------------------------------
"
stringToSign="AWS4-HMAC-SHA256
$fulldate
$shortdate/fr-par/s3/aws4_request
$canonReqSha"
echo $stringToSign
dateKey=$(echo -n "$shortdate" | openssl dgst -sha256 -binary -hmac "AWS4$SCW_SECRET_KEY")
regionKey=$(echo -n "fr-par" | openssl dgst -sha256 -binary -hmac "$dateKey")
serviceKey=$(echo -n "s3" | openssl dgst -sha256 -binary -hmac "$regionKey")
signingKey=$(echo -n "aws4_request" | openssl dgst -sha256 -binary -hmac "$serviceKey")
signature=$(echo -n "$stringToSign" | openssl dgst -sha256 -hmac "$signingKey" | awk '{print $2}')
echo "signature: $signature"
echo "
---------------------------------------
"
# Make request
curl -X PUT \
-H "Content-Type: text/plain" \
-H "x-amz-content-sha256: $file_sha256" \
-H "x-amz-date: $fulldate" \
-H "Authorization: AWS4-HMAC-SHA256" \
-H "Credential: $SCW_ACCESS_KEY/${shortdate}/$REGION/s3/aws4_request" \
-H "SignedHeaders: content-type;host;x-amz-content-sha256;x-amz-date" \
-H "Signature: $signature" \
--data-binary #../import.yml \
"http://$BUCKET.s3.$REGION.scw.cloud/import.yml"
The request is done in clear http just because it's simpler to capture it with network analysis tool (e.g. wireshark) than https.

Related

copy var partial bytes in shell

The following contents in my shell file :
#!/usr/bin/env bash
api_params_with_timestamp="somethingXXXX"
SECRET_KEY="XXXXXXXXXXXXXX"
signature=$(echo -n "$api_params_with_timestamp" \
| openssl dgst -sha256 -hmac "$SECRET_KEY")
echo "signature"=$signature
execute this shell I got
signature=(stdin)= 3eee8d204c83381dddffgggg
I just need "3eee8d204c83381dddffgggg" part , what can I do
so that (stdin)= part would be skip ?!
Edit:
signature=$(echo -n "$api_params_with_timestamp" \
| openssl dgst -sha256 -hmac "$SECRET_KEY" | cut -c10-1000)
not perfect but works !
The openssl-dgst page doesn't provide a direct option to drop the "(stdin)= " portion of the output, but that can be done trivially with cut, grep or sed. For example:
with cut
signature=$(echo -n "$api_params_with_timestamp" \
| openssl dgst -sha256 -hmac "$SECRET_KEY" \
| cut -d ' ' -f2)
with grep
signature=$(echo -n "$api_params_with_timestamp" \
| openssl dgst -sha256 -hmac "$SECRET_KEY" \
| grep -E -o '[^ ]+$')
with sed
signature=$(echo -n "$api_params_with_timestamp" \
| openssl dgst -sha256 -hmac "$SECRET_KEY" \
| sed 's/^.*[[:space:]]//')
Note: you can also remove the line separator characters "\" and use the pipe for that purpose instead, e.g.
signature=$(echo -n "$api_params_with_timestamp" |
openssl dgst -sha256 -hmac "$SECRET_KEY" |
sed 's/^.*[[:space:]]//')
Either way is fine -- up to you.
Example Use/Output
$ api_params_with_timestamp="somethingXXXX"
> SECRET_KEY="XXXXXXXXXXXXXX"
> signature=$(echo -n "$api_params_with_timestamp" |
> openssl dgst -sha256 -hmac "$SECRET_KEY" |
> sed 's/^.*[[:space:]]//')
> echo $signature
46c86ea27df85b8ab9a8c565a344cadd1230f32ff1d773ba0fa5aa1076116d0b

Why is my key not found when sending a JWT assertion to Azure AD?

I'm trying to write a bash script to get access tokens from Microsoft for my app registered in AD. I can't seem to get past this error though:
"error":"invalid_client","error_description":"AADSTS700027: The certificate with identifier used to sign the client assertion is not registered on application. [Reason - The key was not found., Thumbprint of key used by client: '356134 ...
The full gist can be found here: https://gist.github.com/smaring/3a3a6779a809beecc39624aada6e2b88
Here are some of the juicy bits ...
$ openssl pkcs12 -in <your-app>.pfx -out <your-app>.pem
$ openssl x509 -outform der -in <your-app>.pem -out ${PUBLIC_CERT_FILE}
$ openssl rsa -in <your-app>.pem -out ${PRIVATE_KEY_FILE}
x5t="$(sha1sum ${PUBLIC_CERT_FILE} | awk '{print $1;}' | openssl base64 | sed s/\+/-/g |sed 's/\//_/g' | sed -E s/=+$// )"
read -r -d '' HEADER <<EOF
{
"alg": "RS256",
"typ": "JWT",
"x5t": "${x5t}"
}
EOF
HEADER_NO_WHITESPACE=$(echo "${HEADER}" | sed ':a; N; s/[[:space:]]//g; ta')
BASE64_ENCODED_HEADER=$(echo ${HEADER_NO_WHITESPACE} | openssl base64 | sed s/\+/-/g |sed 's/\//_/g' | sed -E s/=+$// )
read -r -d '' PAYLOAD <<EOF
{
"aud": "https: //login.microsoftonline.com/${TENANT_ID}/oauth2/v2.0/token",
"exp": ${exp},
"iss": "${CLIENT_ID}",
"jti": "${jti}",
"nbf": ${nbf},
"sub": "${CLIENT_ID}"
}
EOF
PAYLOAD_NO_WHITESPACE=$(echo "${PAYLOAD}" | sed ':a; N; s/[[:space:]]//g; ta')
BASE64_ENCODED_PAYLOAD=$(echo ${PAYLOAD_NO_WHITESPACE} | openssl base64 | sed s/\+/-/g |sed 's/\//_/g' | sed -E s/=+$// )
SIGNATURE=$( echo -n "${BASE64_ENCODED_HEADER}.${BASE64_ENCODED_PAYLOAD}" | \
openssl dgst -sha256 -binary -sign <(cat ${PRIVATE_KEY_FILE}) | \
openssl base64 | sed s/\+/-/g |sed 's/\//_/g' | sed -E s/=+$// )
CLIENT_ASSERTION="${BASE64_ENCODED_HEADER}.${BASE64_ENCODED_PAYLOAD}.${SIGNATURE}"
curl -s -X POST \
--header \"Content-Type: application/x-www-form-urlencoded\" \
-d \"\
scope=https%3A%2F%2Fgraph.microsoft.com%2F.default&\
grant_type=client_credentials&\
client_assertion_type=urn%3Aietf%3Aparams%3Aoauth%3Aclient-assertion-type%3Ajwt-bearer&\
client_id=${CLIENT_ID}&\
client_assertion=${CLIENT_ASSERTION}\" \
https://login.microsoftonline.com/${TENANT_ID}/oauth2/v2.0/token"
After getting a Java version of this working I figured out that I was not generating the thumbprint properly. The fix was this:
thumbprint=$(openssl x509 -in ${publicCertFile} -fingerprint -noout | awk '{split($0,a,"="); print a[2]}' )
thumbprintHex=$(echo -e "\\x${thumbprint}" | sed 's/:/\\x/g' )
thumbprintHexString=$(echo -e "${thumbprint}" | sed 's/://g' )
x5t=$(echo -ne "${thumbprintHex}" | openssl base64 | sed s/\+/-/g | sed 's/\//_/g' | sed -E s/=+$// )
I have updated the gist to the fully working bash script.
Construct a JWT assertion in bash using client-credential with cert to get an access token from Microsoft Azure Directory --> https://gist.github.com/smaring/3a3a6779a809beecc39624aada6e2b88

Call to Tuya API via bash

I am having an hard time calling the Tuya API while using curl.
Tuya requires to generate a signature as following:
HMAC-SHA256(client_id + t, secret).
I built a small script that does exactly what Tuya asks. I have also double checked by trying to generate the signature using the same client_id, t and secret that are in their documentation as example, and the generated signature matches what the documentation says.
client_id is a pre-assigned value
t is the timestamp in 13 digits (and here I think is where the error is)
secret is a pre-assigned value
Once the signature is built it needs to be used via curl in a POST call, but Tuya keeps refusing the signature with the following error:
{"code":1004,"msg":"sign invalid","success":false,"t":1664314067553}
Now, I think that the issue is the timing.
In order for my script to generate the signature few milliseconds are required and when the value of t gets passed to curl it won't match with the execution of curl (of course). Here's my code:
t=($(($(date +%s%N)/1000000))); sign1=$(echo -n "yyr8hxxxxxxxxd4mji$t" | openssl dgst -sha256 -hmac "cc75fd7xxxxxxxxx63d032b" | awk '{print$2}') && sign2=$(echo ${sign1^^}) ; curl --request POST "https://openapi.tuyaeu.com/v1.0/iot-03/devices/717715xxxxxxx520/commands" --header "sign_method: HMAC-SHA256" --header "client_id: yyr8hxxxxxxxxd4mji" --header "t: t" --header "mode: cors" --header "sign: $sign2" --header "access_token: cc75fd7xxxxxxxxx63d032b" --data "{"commands":[{"code":"switch_1","value":true}]}"
I've of course already tried to use && to execute all commands together but there has been no change. Does someone have any idea?
Your sign is invalid.
Declare your variables:
ClientID="replace_with_you_client_Id yyr8hxxxxxxxxd4mji"
ClientSecret="replace_with_you_client_secret cc75fd7xxxxxxxxx63d032b"
Device="replace_with_your_device 717715xxxxxxx520"
First you have to get an access_token:
AccessToken=$(t=$(date +%s%N |sed "s/......$//g"); curl -sSLkX GET "https://openapi.tuyaeu.com/v1.0/token?grant_type=1" -H "sign_method: HMAC-SHA256" -H "client_id: $ClientID" -H "t: $t" -H "mode: cors" -H "Content-Type: application/json" -H "sign: $(echo -en "${ClientID}${t}GET\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\n\n/v1.0/token?grant_type=1" | openssl dgst -sha256 -hmac "$ClientSecret" | tr '[:lower:]' '[:upper:]' |sed "s/.* //g")" -H "access_token: " | sed "s/.*\"access_token\":\"//g" |sed "s/\".*//g")`
Then you need to calculate the good "sign". You need you ClientId, the timestamp, the access_token, the method, the encoded_body, the url:
TimeStamp=$(date +%s%N |sed "s/......$//g")
METHOD='POST'
BODY='{"commands":[{"code":"switch_1","value":true}]}'
encodedBody=$(echo -n "$BODY" | openssl dgst -sha256 | sed "s/.*[ ]//g")
URL="/v1.0/iot-03/devices/$Device/commands"
Calculate sign:
SIGN=$(echo -n "$ClientID${AccessToken}${TimeStamp}${METHOD}
$encodedBody
$URL" | openssl dgst -sha256 -hmac "$ClientSecret" | tr '[:lower:]' '[:upper:]' |sed "s/.* //g")`
Or in one line:
SIGN=$(echo -en "$ClientID${AccessToken}${TimeStamp}${METHOD}\n$encodedBody\n\n$URL" | openssl dgst -sha256 -hmac "$ClientSecret" | tr '[:lower:]' '[:upper:]' |sed "s/.* //g")
Then send the request:
curl -sSLkX $METHOD "https://openapi.tuyaeu.com$URL" -H "sign_method: HMAC-SHA256" -H "client_id: $ClientID" -H "t: $TimeStamp" -H "mode: cors" -H "Content-Type: application/json" -H "sign: $SIGN" -H "access_token: $AccessToken" -d "$BODY"
I had a use case where I wanted to pull the electric consumption of my devices from the Tuya smart plugs. I followed the Tuya API creation instruction from https://github.com/jasonacox/tuyapower and then encountered the same issue as you, getting {"code":1004,"msg":"sign invalid"," ... forever.
The original answer from bobolecoco was not working for me either. Using Tuya's documentation on https://developer.tuya.com/en/docs/iot/singnature?id=Ka43a5mtx1gsc I figured out that the generated sign was invalid due to line breaks, and used printf instead of echo. See below the code that's easy to debug and that works for me in bash 5.1.16.
# Set debug value to true or false to (de)activate output
debug=true
# Declare constants
ClientID="<<ENTER CLIENT ID HERE>>"
ClientSecret="<<ENTER CLIENT SECRET HERE>>"
BaseUrl="https://openapi.tuyaeu.com"
EmptyBodyEncoded="e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
tuyatime=`(date +%s)`
tuyatime=$tuyatime"000"
if ($debug) then echo Tuyatime is now $tuyatime; fi;
# Get Access Token
URL="/v1.0/token?grant_type=1"
StringToSign="${ClientID}${tuyatime}GET\n${EmptyBodyEncoded}\n\n${URL}"
if ($debug) then echo StringToSign is now $StringToSign; fi;
AccessTokenSign=$(printf $StringToSign | openssl sha256 -hmac "$ClientSecret" | tr '[:lower:]' '[:upper:]' |sed "s/.* //g")
if ($debug) then echo AccessTokenSign is now $AccessTokenSign; fi;
AccessTokenResponse=$(curl -sSLkX GET "$BaseUrl$URL" -H "sign_method: HMAC-SHA256" -H "client_id: $ClientID" -H "t: $tuyatime" -H "mode: cors" -H "Content-Type: application/json" -H "sign: $AccessTokenSign")
if ($debug) then echo AccessTokenResponse is now $AccessTokenResponse; fi;
AccessToken=$(echo $AccessTokenResponse | sed "s/.*\"access_token\":\"//g" |sed "s/\".*//g")
if ($debug) then echo Access token is now $AccessToken; fi;
# Send Device status request
URL="/v1.0/iot-03/devices/status?device_ids=<<ENTER DEVICE IDs HERE, COMMA SEPARATED>>"
StringToSign="${ClientID}${AccessToken}${tuyatime}GET\n${EmptyBodyEncoded}\n\n${URL}"
if ($debug) then echo StringToSign is now $StringToSign; fi;
RequestSign=$(printf $StringToSign | openssl sha256 -hmac "$ClientSecret" | tr '[:lower:]' '[:upper:]' |sed "s/.* //g")
if ($debug) then echo RequestSign is now $RequestSign; fi;
RequestResponse=$(curl -sSLkX GET "$BaseUrl$URL" -H "sign_method: HMAC-SHA256" -H "client_id: $ClientID" -H "t: $tuyatime" -H "mode: cors" -H "Content-Type: application/json" -H "sign: $RequestSign" -H "access_token: $AccessToken")
if ($debug) then echo RequestResponse is now $RequestResponse; fi;

Uploading to Amazon S3 via curl route

I am trying to set up a file upload REST API via Spring Boot.
I currently have a list/GET method curl http://localhost:8080/api/aws/s3/list which returns a list of objects that currently exist in the bucket.
For upload, I have been trying:
curl -F "data=#test.txt" http://localhost:8080/api/aws/s3/upload -i
Which produces:
HTTP/1.1 100 Continue
HTTP/1.1 200 OK
Date: Sun, 25 Jun 2017 23:28:36 GMT
X-Application-Context: application
Content-Type: application/json;charset=utf-8
Transfer-Encoding: chunked
But when I look at the bucket, it hasn't been updated with the new file.
Would this be a permissions issue on AWS? Only my account has read and write access. The user and group I created have admin privileges. I didn't add a Bucket Policy. This is my CORS configuration:
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>POST</AllowedMethod>
<AllowedMethod>DELETE</AllowedMethod>
<MaxAgeSeconds>3000</MaxAgeSeconds>
<AllowedHeader>Authorization</AllowedHeader>
</CORSRule>
</CORSConfiguration>
Here is the upload section of my S3 Controller in spring:
#RequestMapping(value = "/upload", method = RequestMethod.POST)
public List<PutObjectResult> upload(#RequestParam("file") MultipartFile[] multipartFiles) {
return s3Wrapper.upload(multipartFiles);
}
You should specify your using bucket name and some parameters for S3. And I guess it's PUT, not POST. There are several command line samples in the internet.
file=/path/to/file/to/upload.tar.gz
bucket=your-bucket
resource="/${bucket}/${file}"
contentType="application/x-compressed-tar"
dateValue=`date -R`
stringToSign="PUT\n\n${contentType}\n${dateValue}\n${resource}"
s3Key=xxxxxxxxxxxxxxxxxxxx
s3Secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -X PUT -T "${file}" \
-H "Host: ${bucket}.s3.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Content-Type: ${contentType}" \
-H "Authorization: AWS ${s3Key}:${signature}" \
https://${bucket}.s3.amazonaws.com/${file}
Uploading to S3 in 18 lines of Shell (used to upload builds for http://soltrader.net)
Uploading to S3 in Bash
#run this code on ec2 linux with s3 write role
TOKEN=`curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
xamztoken=`curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/ec2files3 | jq -r ".Token"`
yyyymmdd=`date +%Y%m%d`
s3Bucket="testfiles3bc"
bucketLocation="us-east-1"
s3SecretKey=`curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/ec2files3 | jq -r '.SecretAccessKey'`
s3AccessKey=`curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/ec2files3 | jq -r ".AccessKeyId"`
endpoint="testfiles3bc.s3.amazonaws.com"
fileName="image.gif"
contentLength=`cat ${fileName} | wc -c`
contentHash=`openssl sha -sha256 -hex ${fileName} | sed 's/.* //'`
contentType=`file -b --mime-type $fileName`
b64=`openssl md5 -binary "$fileName" | openssl base64`
acl="private"
date=`date -u +%Y%m%dT%H%M%SZ`
expdate_s="2022-12-30T12:00:00.000Z"
region="us-east-1"
p=$(cat <<POLICY | openssl base64 | tr -d \\n
{ "expiration": "${expdate_s}T12:00:00.000Z",
"conditions": [
{"acl": "$acl" },
{"bucket": "$s3Bucket" },
["starts-with", "\$key", ""],
["starts-with", "$contentType", "image/"],
{"x-amz-date": "$date" },
{"content-md5": "$b64" },
{"x-amz-credential": "${s3AccessKey}/${yyyymmdd}/${region}/s3/aws4_request" },
{"x-amz-security-token": "${xamztoken}" },
{"x-amz-algorithm": "AWS4-HMAC-SHA256" }
]
}
POLICY
)
stringToSign=$p
echo "----------------- canonicalRequest --------------------"
echo -e ${canonicalRequest}
echo "----------------- stringToSign --------------------"
echo -e ${stringToSign}
echo "-------------------------------------------------------"
# calculate the signing key
DateKey=`echo -n "${yyyymmdd}" | openssl sha -sha256 -hex -hmac "AWS4${s3SecretKey}" | sed 's/.* //'`
DateRegionKey=`echo -n "${bucketLocation}" | openssl sha -sha256 -hex -mac HMAC -macopt hexkey:${DateKey} | sed 's/.* //'`
DateRegionServiceKey=`echo -n "s3" | openssl sha -sha256 -hex -mac HMAC -macopt hexkey:${DateRegionKey} | sed 's/.* //'`
SigningKey=`echo -n "aws4_request" | openssl sha -sha256 -hex -mac HMAC -macopt hexkey:${DateRegionServiceKey} | sed 's/.* //'`
# then, once more a HMAC for the signature
signature=`echo -en ${p} | openssl sha -sha256 -hex -mac HMAC -macopt hexkey:${SigningKey} | sed 's/.* //'`
key_and_sig_args="-F X-Amz-Credential=${s3AccessKey}/${yyyymmdd}/${region}/s3/aws4_request -F X-Amz-Algorithm=AWS4-HMAC-SHA256 -F X-Amz-Signature=$signature -F X-Amz-Date=${date}"
curl -v \
-F key=$fileName \
-F acl=$acl \
$key_and_sig_args \
-F "content-md5= ${b64}" \
-F "Policy=$p" \
-F "X-Amz-Security-Token= ${xamztoken}" \
-F "file=#$fileName" \
https://${s3Bucket}.s3.amazonaws.com/
It was because the parameter I set in my controller was file but in my curl route I kept using data

Download private file from S3 using bash

I am trying to get the following bash script to work (copied from http://curl.haxx.se/mail/archive-2014-10/0006.html#replies):
#!/bin/sh
file=path/to/file
bucket=your-bucket
resource="/${bucket}/${file}"
contentType="application/x-compressed-tar"
dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`"
stringToSign="GET
${contentType}
${dateValue}
${resource}"
s3Key=xxxxxxxxxxxxxxxxxxxx
s3Secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
signature=`/bin/echo -n "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -H "Host: ${bucket}.s3.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Content-Type: ${contentType}" \
-H "Authorization: AWS ${s3Key}:${signature}" \
https://${bucket}.s3.amazonaws.com/${file}
I am getting a SignatureDoesNotMatch error no matter what I do.
Any ideas on how to fix this will be greatly appreciated.
After way too much time spent on this I finally got it to work:
This line:
signature=`/bin/echo -n "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64`
is missing an 'e':
signature=`/bin/echo -en "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64`
In other words, characters weren't being escaped before the string was signed.
As an aside, I also learned that for get requests, the content type is meaningless.
Using various answers in this thread, I converted it into a handy s3get bash function:
#!/bin/bash
#usage - s3get writes the specified object to stdout
# s3get <bucket/key> [region]
#set these in your environment/profile (NOT HERE)
AWS_ACCESS_KEY=""
AWS_SECRET_KEY=""
#example usage
s3get my-bucket/a/path/to/my/file > /tmp/file
function s3get {
#helper functions
function fail { echo "$1" > /dev/stderr; exit 1; }
#dependency check
if ! hash openssl 2>/dev/null; then fail "openssl not installed"; fi
if ! hash curl 2>/dev/null; then fail "curl not installed"; fi
#params
path="${1}"
bucket=$(cut -d '/' -f 1 <<< "$path")
key=$(cut -d '/' -f 2- <<< "$path")
region="${2:-us-west-1}"
#load creds
access="$AWS_ACCESS_KEY"
secret="$AWS_SECRET_KEY"
#validate
if [[ "$bucket" = "" ]]; then fail "missing bucket (arg 1)"; fi;
if [[ "$key" = "" ]]; then fail "missing key (arg 1)"; fi;
if [[ "$region" = "" ]]; then fail "missing region (arg 2)"; fi;
if [[ "$access" = "" ]]; then fail "missing AWS_ACCESS_KEY (env var)"; fi;
if [[ "$secret" = "" ]]; then fail "missing AWS_SECRET_KEY (env var)"; fi;
#compute signature
contentType="text/html; charset=UTF-8"
date="`date -u +'%a, %d %b %Y %H:%M:%S GMT'`"
resource="/${bucket}/${key}"
string="GET\n\n${contentType}\n\nx-amz-date:${date}\n${resource}"
signature=`echo -en $string | openssl sha1 -hmac "${secret}" -binary | base64`
#get!
curl -H "x-amz-date: ${date}" \
-H "Content-Type: ${contentType}" \
-H "Authorization: AWS ${access}:${signature}" \
"https://s3-${region}.amazonaws.com${resource}"
}
Tested on OSX and Ubuntu. Saved in this Github gist.
The TS asked for a working SHA-1 version of the script. However, SHA-1 is outdated and Amazon has datacenters that only accept SHA-256 encryption, hereby the download script that can be used for all S3 datacenters:
It also follows HTTP 307 redirects.
#!/bin/sh
#USAGE:
# download-aws.sh <bucket> <region> <source-file> <dest-file>
set -e
s3Key=xxxxxxxxxxxxxxxxxxxx
s3Secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
file=$3
bucket=$1
host="${bucket}.s3.amazonaws.com"
resource="/${file}"
contentType="text/plain"
dateValue="`date +'%Y%m%d'`"
X_amz_date="`date +'%Y%m%dT%H%M%SZ'`"
X_amz_algorithm="AWS4-HMAC-SHA256"
awsRegion=$2
awsService="s3"
X_amz_credential="$s3Key%2F$dateValue%2F$awsRegion%2F$awsService%2Faws4_request"
X_amz_credential_auth="$s3Key/$dateValue/$awsRegion/$awsService/aws4_request"
signedHeaders="host;x-amz-algorithm;x-amz-content-sha256;x-amz-credential;x-amz-date"
contentHash="e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
HMAC_SHA256_asckey () {
var=`/bin/echo -en $2 | openssl sha256 -hmac $1 -binary | xxd -p -c256`
echo $var
}
HMAC_SHA256 () {
var=`/bin/echo -en $2 | openssl dgst -sha256 -mac HMAC -macopt hexkey:$1 -binary | xxd -p -c256`
echo $var
}
REQUEST () {
canonicalRequest="GET\n$resource\n\n"\
"host:$1\n"\
"x-amz-algorithm:$X_amz_algorithm""\n"\
"x-amz-content-sha256:$contentHash""\n"\
"x-amz-credential:$X_amz_credential""\n"\
"x-amz-date:$X_amz_date""\n\n"\
"$signedHeaders\n"\
"$contentHash"
#echo $canonicalRequest
canonicalHash=`/bin/echo -en "$canonicalRequest" | openssl sha256 -binary | xxd -p -c256`
stringToSign="$X_amz_algorithm\n$X_amz_date\n$dateValue/$awsRegion/s3/aws4_request\n$canonicalHash"
#echo $stringToSign
s1=`HMAC_SHA256_asckey "AWS4""$s3Secret" $dateValue`
s2=`HMAC_SHA256 "$s1" "$awsRegion"`
s3=`HMAC_SHA256 "$s2" "$awsService"`
signingKey=`HMAC_SHA256 "$s3" "aws4_request"`
signature=`/bin/echo -en $stringToSign | openssl dgst -sha256 -mac HMAC -macopt hexkey:$signingKey -binary | xxd -p -c256`
#echo signature
authorization="$X_amz_algorithm Credential=$X_amz_credential_auth,SignedHeaders=$signedHeaders,Signature=$signature"
result=$(curl --silent -H "Host: $1" -H "X-Amz-Algorithm: $X_amz_algorithm" -H "X-Amz-Content-Sha256: $contentHash" -H "X-Amz-Credential: $X_amz_credential" -H "X-Amz-Date: $X_amz_date" -H "Authorization: $authorization" https://${1}/${file} -o "$2" --write-out "%{http_code}")
if [ $result -eq 307 ]; then
redirecthost=`cat $2 | sed -n 's:.*<Endpoint>\(.*\)</Endpoint>.*:\1:p'`
REQUEST "$redirecthost" "$2"
fi
}
REQUEST "$host" "$4"
Tested on Ubuntu
If someone knows a solution to remove the HMAC-ASCII step, you're welcome to reply. I got this only working in this way.
It required minor adjustment, but the following lines works well
#!/bin/sh
file=path/to/file
bucket=your-bucket
resource="/${bucket}/${file}"
contentType="application/x-compressed-tar"
dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`"
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key=xxxxxxxxxxxxxxxxxxxx
s3Secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
signature=`/bin/echo -en "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -H "Host: ${bucket}.s3.amazonaws.com" -H "Date: ${dateValue}" -H "Content-Type: ${contentType}" -H "Authorization: AWS ${s3Key}:${signature}" https://${bucket}.s3.amazonaws.com/${file}
bucket=your-bucket-name
contentType="text/plain"
dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`"
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key=xxxxxx
s3Secret=xxxxx
signature=`/bin/echo -en "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64`
file1=file-name
resource1="/${bucket}/${file1}"
curl -H "Date: ${dateValue}" -H "Content-Type: ${contentType}" -H "Authorization: AWS ${s3Key}:${signature}" "https://s3-us-west-2.amazonaws.com/${resource1}" -o "file-name-to-save-the-output"
I was getting errors in the actual answer. This works for me. This will get the file as what it is and not as string.

Resources