How to convert JSON Web Key Set (JWKS) public keys to PEM file using BASH? - bash

Let's say you use AWS and you want to use ID based authentication using Cognito.
Then aws provides you with a public key you can verify the cognito payload with.
Let's also assume you don't want or cannot use any fancy libraries like jose since you are locked in a highly constrained environment.
The way to go is a BASH script that would make good old Brian Kernighan
proud.
You have to understand the encoding first.
Base64Url needs to be translated to Base64.
This is achieved using padding characters =
If the number of characters is divisible by 4 you don't need padding.
This relates to the binary digit representation.
Once that is taken care of you can translate Base64 to binary.
But how do I convert a JWKS/JWT to a PEM file this using BASH and BASH-programs only?

This is the solution I came up with following the officual documentation and sources listed below.
https://aws.amazon.com/premiumsupport/knowledge-center/decode-verify-cognito-json-token/
Please adapt input url or use json token directly and make sure you have jq installed.
Descriptive help is given in the function as comments.
Tested successfully on Ubuntu 18.04 and AmazonLinux2 (CentOS) as of 2020-04-28
#!/usr/bin/env bash
set -e
# FUNCTIONS
decodeBase64UrlUInt() { #input:base64UrlUnsignedInteger
local binaryDigits paddedStr
case $(( ${#1} % 4 )) in
2) paddedStr="$1==" ;;
3) paddedStr="$1=" ;;
*) paddedStr="$1" ;;
esac
binaryDigits=$( \
echo -n "$paddedStr" \
| tr '_-' '/+' \
| openssl enc -d -a -A \
| xxd -b -g 0 \
| cut -d ' ' -f 2 \
| paste -s -d '' \
)
echo "ibase=2; obase=A; $binaryDigits" | bc
# openssl enc:encoding; -d=decrypt; -a=-base64; -A=singleLineBuffer
# xxd "make-hexdump": -b=bits; -g=groupsize
# cut -d=delimiter; -f=field
# paste -s=serial|singleFile; -d=delimiter
}
base64UrlToHex() { #input:base64UrlString
local hexStr paddedStr
case $(( ${#1} % 4 )) in
2) paddedStr="$1==" ;;
3) paddedStr="$1=" ;;
*) paddedStr="$1" ;;
esac
hexStr=$( \
echo -n "$paddedStr" \
| tr '_-' '/+' \
| base64 -d \
| xxd -p -u \
| tr -d '\n' \
)
echo "$hexStr"
# base64 -d=decode
# xxd -p=-plain=continuousHexDump; -u=upperCase
# tr -d=delete
}
asn1Conf() { #input:hexStrPlainUpperCase
local e="$1"
local n="$2"
echo "
asn1 = SEQUENCE:pubkeyinfo
[pubkeyinfo]
algorithm = SEQUENCE:rsa_alg
pubkey = BITWRAP,SEQUENCE:rsapubkey
[rsa_alg]
algorithm = OID:rsaEncryption
parameter = NULL
[rsapubkey]
n = INTEGER:0x$n
e = INTEGER:0x$e
" | sed '/^$/d ; s/^ *//g' \
| openssl asn1parse \
-genconf /dev/stdin \
-out /dev/stdout \
| openssl rsa \
-pubin \
-inform DER \
-outform PEM \
-in /dev/stdin \
-out /dev/
# sed /^$/d=removeEmptyLines; /^ */=removeLeadingSpaces
}
main() {
local e n hexArr
local jwksUrl="$1"
local jwkJson=$(curl -sSSL $jwksUrl)
local kidList=$(jq -r '.keys[].kid' <<< "$jwkJson")
for keyId in $kidList; do
n=$(jq -r ".keys[] | select(.kid == \"$keyId\") | .n" <<< "$jwkJson")
e=$(jq -r ".keys[] | select(.kid == \"$keyId\") | .e" <<< "$jwkJson")
echo -e "\n$keyId"
# decodeBase64UrlUInt "$e"
# decodeBase64UrlUInt "$n"
asn1Conf $(base64UrlToHex "$e") $(base64UrlToHex "$n")
done
}
# MAIN
main 'https://cognito-idp.{region}.amazonaws.com/{userPoolId}/.well-known/jwks.json'
exit 0
Special thanks to:
Yury Oparin
https://www.yuryoparin.com/2014/05/base64url-in-bash.html
Cédric Deltheil
https://github.com/Moodstocks/moodstocks-api-clients/blob/master/bash/base64url.sh
Alvis Tang
https://gist.github.com/alvis/89007e96f7958f2686036d4276d28e47

Here are some options:
Either ignore base64 -d complaining of truncated input:
<<<'SGVsbG8geW91Cg' base64 -d 2>/dev/null ||:
Or fix the base64 padding with Bash before decoding:
base64URL='SGVsbG8geW91Cg'
printf -v pad_space '%*s' $((${#base64URL}%4)) ''
padded_base64="$base64URL${pad_space// /=}"
<<<"$padded_base64" base64 -d

Related

Writing a comparison BATCH file to verify sha256sum to released code

Trying to write a script that takes 2 arguments ($1 and $2) one to represent the $hash and the $file_name.
I am trying to utilize jq to parse the required data to download and compare PASS or FAIL.
I see to be stuck trying to think this out.
Here is my code
#!/usr/bin/env sh
#
# Sifchain shasum check (revised).
#
# $1
hash_url=$( curl -R -s https://api.github.com/repos/Sifchain/sifnode/releases | jq '.[] | select(.name=="v0.10.0-rc.4")' | jq '.assets[]' | jq 'select(.name=="sifnoded-v0.10.0-rc.4-linux-amd64.zip.sha256")' | jq '.browser_download_url' | xargs $1 $2 )
echo $hash_url
# $2
hash=$( curl -s -L $hash_url | jq'.$2')
file_name=$(curl -R -s https://api.github.com/repos/Sifchain/sifnode/releases | jq '.[] | .name')
#
#
echo $hash | sha256sum
echo $file_name | sha256sum #null why?
echo "\n"
## version of the release $1, and the hash $2
## sha256 <expected_sha_256_sum> <name_of_the_file>
sha256() {
if echo "$1 $2" #| sha256sum -c --quiet
then
echo pass $1 $2
exit 0
else
echo FAIL $1 $2
exit 1
fi
}
# Invoke sha256
sha256 $hash_url $file_name
Ideally this should work for any comparison of hash with correct file, pulling the 2 parameters when the BASH script is invoked.
I can suggest the following corrections/modifications:
#!/bin/bash
#sha file
SHA_URL=$(curl -R -s https://api.github.com/repos/Sifchain/sifnode/releases | \
jq --arg VERSION v0.10.0-rc.4 -r \
'.[] | select(.name==$VERSION) | .assets[] | select(.name |test("\\.sha256$")) | .browser_download_url')
SHA_VALUE=$(curl -s -L $SHA_URL| tr 1 2)
FILENAME=$(curl -R -s https://api.github.com/repos/Sifchain/sifnode/releases | \
jq --arg VERSION v0.10.0-rc.4 -r \
'.[] | select(.name==$VERSION) | .assets[] | select(.content_type =="application/zip") | .name')
#added just for testing, I'm assuming you have the files locally allready
FILEURL=$(curl -R -s https://api.github.com/repos/Sifchain/sifnode/releases | \
jq --arg VERSION v0.10.0-rc.4 -r \
'.[] | select(.name==$VERSION) | .assets[] | select(.content_type =="application/zip") | .browser_download_url')
wget --quiet $FILEURL -O $FILENAME
echo $SHA_VALUE $FILENAME | sha256sum -c --quiet >/dev/null 2>&1
RESULT=$?
if [ $RESULT -eq 0 ]; then
echo -n "PASS "
else
echo -n "FAIL "
fi
echo $SHA_VALUE $FILENAME
exit $RESULT
Notes:
jq
--arg VERSION v0.10.0-rc.4 creates a "variable" to be used in the script
-r - raw output, strings are not quoted
test("\\.sha256$") - regular expresion, used to search for a generic sha256, so you don't have to hardcode the full name
select(.content_type =="application/zip") - I'm assuming that's the file you are searching for
wget is used just for demo purpose, to download the file, I'm assuming you already have the file on your machine
sha256sum -c --quiet >/dev/null 2>&1 - redirecting to /dev/null is necessary because in case of error sha256sum is not quiet

Bash OpenSSL is not equal to php openssl_encrypt value

I am trying to reach the same value of encryption in bash as it is from php , with no success .
Here is my php script
$message ='{"coupon_key":"011205358365345679","location_id":"288","device_key":"test_1234"}';
$key = "password";
$iv = base64_decode("dU+AyWweQYhAlGKLaxoS9w==");
$base64_iv = base64_encode($iv);
$base64_ev = base64_encode($encrypted_value);
$encrypted_value = openssl_encrypt(
$message,
'AES-256-CBC',
$key,
OPENSSL_RAW_DATA|OPENSSL_NO_PADDING,
$iv
);
$encode = base64_encode($encrypted_value);
//Zi7UcBwqM+gKZC9iZPgF3UHBXLUg1+Age/1+kRIfcstYBgGfYm7N1qCIFnm6jGn4AGQph/Q6hKjR1hYBT0wkJv8c8qFrHHZNUuraMfSRH3s=
$mac = hash_hmac('sha256', $base64_iv . $encode, $key);
echo '{"iv":"'.$base64_iv.'","value":"'.$encode.'","mac":"'.$mac.'"}'."<br>";
//{"iv":"dU+AyWweQYhAlGKLaxoS9w==","value":"Zi7UcBwqM+gKZC9iZPgF3UHBXLUg1+Age/1+kRIfcstYBgGfYm7N1qCIFnm6jGn4AGQph/Q6hKjR1hYBT0wkJv8c8qFrHHZNUuraMfSRH3s=","mac":"97fb6f4226a917322c7361af1d9b2949ad96691c1fc1a7f1c8379c71ae19f356"}
$retString2 = base64_encode('{"iv":"'.$base64_iv.'","value":"'.$encode.'","mac":"'.$mac.'"}');
echo $retString2;
//eyJpdiI6ImRVK0F5V3dlUVloQWxHS0xheG9TOXc9PSIsInZhbHVlIjoiWmk3VWNCd3FNK2dLWkM5aVpQZ0YzVUhCWExVZzErQWdlLzEra1JJZmNzdFlCZ0dmWW03TjFxQ0lGbm02akduNEFHUXBoL1E2aEtqUjFoWUJUMHdrSnY4YzhxRnJISFpOVXVyYU1mU1JIM3M9IiwibWFjIjoiOTdmYjZmNDIyNmE5MTczMjJjNzM2MWFmMWQ5YjI5NDlhZDk2NjkxYzFmYzFhN2YxYzgzNzljNzFhZTE5ZjM1NiJ9
and here is my bash version
password="password";
passwordhex=$(echo "$password" | xxd -c 256 -ps) ;
# iv2=$(hexdump -n 16 -e '4/4 "%08X" 1 "\n"' /dev/random) ;
iv=$(printf 'dU+AyWweQYhAlGKLaxoS9w==' | base64 -d )
biv=$(printf "$iv" | base64) ;
hexiv=$(printf "$iv" | xxd -c 256 -ps) ;
eString=$(printf '{"coupon_key":"011205358365345678","location_id":"288","device_key":"test_1234"}');
tttw=$(printf "$eString" | openssl aes-256-cbc -e -nosalt -a -A -K "$passwordhex" -iv "$hexiv") ;
echo "$tttw" ;
printf "${biv}${tttw}" |openssl dgst -sha256 -hmac abc -macopt hexkey:"$passwordhex" | sed 's/^.* //';
macopt2=$(printf "${biv}${tttw}" |openssl dgst -sha256 -hmac abc -macopt hexkey:"$passwordhex" | sed 's/^.* //');
echo "$macopt2"
finalString="{'iv':'$biv','value':'$tttw','mac':'$macopt2'}";
echo "$finalString";
sendHash=$(printf "$finalString" | base64 -w 0)
echo "$sendHash"
Starting from
tttw=$(echo -n $eString | openssl aes-256-cbc -e -nosalt -a -A -K $passwordhex -iv $hexiv) ;
it goes wrong , because $tttw is returning then
bKG5quB9/YQUsmlFvDHq2H+AfNGQuDfVztyi0dd5hCY7hLfaACnjD8SWlwqy0yy4hXUZSA2YcTXej/xtMg9vqEpoO6CDw9hk7+tUcYOOV5aOdVBnSLowmEllHt0JfjdE
instead of Zi7UcBwqM+gKZC9iZPgF3UHBXLUg1+Age/1+kRIfcstYBgGfYm7N1qCIFnm6jGn4AGQph/Q6hKjR1hYBT0wkJv8c8qFrHHZNUuraMfSRH3s=
Can anybody maybe see what option is going wrong here?
Note1: I have updated the bash script to use prinf instead of echo -n
Note 2 : Seems on my newer machine I get "hex string is too short, padding with zero bytes to length" from the line
tttw=$(printf $eString | openssl aes-256-cbc -e -nosalt -a -A -K $passwordhex -iv $hexiv) ;
Looks like the issue is near the very top:
passwordhex=$(echo "$password" | xxd -c 256 -ps) ;
So essentially the output of echo "$password" is password\n.
Using echo like this will append a line feed to the input for xxd which I assume you already know since the original script was packed with echo -n. I mentioned in the comments that printf is more portable. The accepted way to use printf with a variable is like this:
passwordhex=$(printf '%s' "$password" | xxd -c 256 -ps) ;
The reason for this is that if your variable has a format identifier (like %s), it would change the output unexpectedly. This would have also showed up on shellcheck.
With this one change, here is the output:
Zi7UcBwqM+gKZC9iZPgF3UHBXLUg1+Age/1+kRIfcssepjJ8+wUjTDAjPUMkGA+eF9EL284iD5UIzA+REyhMWLWbUJpPltHFk1+lhQyVlUXXVTw0FFV1G+iQfEWhbyg4
484123c33b54e446c61120112955cd15f3592f42e737c9fa24db266cdec954a2
484123c33b54e446c61120112955cd15f3592f42e737c9fa24db266cdec954a2
{'iv':'dU+AyWweQYhAlGKLaxoS9w==','value':'Zi7UcBwqM+gKZC9iZPgF3UHBXLUg1+Age/1+kRIfcssepjJ8+wUjTDAjPUMkGA+eF9EL284iD5UIzA+REyhMWLWbUJpPltHFk1+lhQyVlUXXVTw0FFV1G+iQfEWhbyg4','mac':'484123c33b54e446c61120112955cd15f3592f42e737c9fa24db266cdec954a2'}
eydpdic6J2RVK0F5V3dlUVloQWxHS0xheG9TOXc9PScsJ3ZhbHVlJzonWmk3VWNCd3FNK2dLWkM5aVpQZ0YzVUhCWExVZzErQWdlLzEra1JJZmNzc2Vwako4K3dValREQWpQVU1rR0ErZUY5RUwyODRpRDVVSXpBK1JFeWhNV0xXYlVKcFBsdEhGazErbGhReVZsVVhYVlR3MEZGVjFHK2lRZkVXaGJ5ZzQnLCdtYWMnOic0ODQxMjNjMzNiNTRlNDQ2YzYxMTIwMTEyOTU1Y2QxNWYzNTkyZjQyZTczN2M5ZmEyNGRiMjY2Y2RlYzk1NGEyJ30=

AES (aes-cbc-256) encryption/decryption with openssl expected output truncated

I wrote a script. Please pardon that I am not an expert in scripting.
Upon deciphering, the results gotten truncated.
[Message in text]: 0123456789abcdefghijklmnopqrstuvwxyz
message_input in hex: 303132333435363738396162636465666768696a6b6c6d6e6f707172737475767778797a0a
key: 788a1ca0bf1ab80f092841aabd77793f
hex string is too short, padding with zero bytes to length
c19f83afc1160ce81b0fc9906d513693386ccdd313b0f2884c698411441054e8
ciphered text: c19f83afc1160ce81b0fc9906d513693386ccdd313b0f2884c698411441054e8
IV: 7ecd3d63a8b74bb2f80d71a1c9d43359
deciphering ...
hex string is too short, padding with zero bytes to length
key: 788a1ca0bf1ab80f092841aabd77793f
iv: 7ecd3d63a8b74bb2f80d71a1c9d43359
answer: 30313233343536373839616263646566
Deciphered Message in hex: 30313233343536373839616263646566
deciphered text: 0123456789abcdef
The recovered deciphered text: 0123456789abcdef, ghijklmnopqrstuvwxyz gets truncated. This is supposed to be AES-CBC. Is there an option I did not turn on?
Here is the ciphering:
IV=$(openssl rand -hex 16)
get_key_for_ciphering; # key_for_ciphering gets populated
message_input=$(echo -n "${message_input//[[:space:]]/}") # remove spaces
echo "message_input in hex: "$message_input
echo "key": $key_for_ciphering;
ANS=$(echo "0: $message_input" | xxd -r | openssl enc -aes-256-cbc -iv $IV -K "$key_for_ciphering" | xxd -p)
ANS=$(echo -n "${ANS//[[:space:]]/}") # remove spaces
Here is the deciphering (message_input=$ANS):
get_key_for_ciphering; # key_for_ciphering gets populated
ANS=$(echo "0: $message_input" | xxd -r | openssl enc -aes-256-cbc -d -nopad -nosalt -K "$key_for_ciphering" -iv $IV | xxd -p) # -nopad -nosalt
Focusing in your question, the problem is in the xxd command. When converting a hex string to binary with xxd -r, you have to use the -p to tell xxd that is a plain hex string (no line breaks).
When converting back to hex with xxd -p, line breaks are added every 32 bytes. Unfortunately, xxd doesn't provide a flag to not include line breaks (you could use -c to set the number of columns, but it's limited to a max number). There are many option to remove line breaks, but one is appending a | tr -d '\n' to your command, as shown in the example below.
IV=$(openssl rand -hex 16)
key_for_ciphering=$(openssl rand -hex 16)
message_input="303132333435363738396162636465666768696a6b6c6d6e6f707172737475767778797a0a"
message_input=$(echo -n "${message_input//[[:space:]]/}") # remove spaces
echo "Message: $message_input"
echo "Key: $key_for_ciphering"
echo "IV: $IV"
ANS=$(echo "0: $message_input" | xxd -r -p | openssl enc -aes-256-cbc -iv $IV -K "$key_for_ciphering" | xxd -p | tr -d '\n')
ANS=$(echo -n "${ANS//[[:space:]]/}") # remove spaces
echo "Encrypted: $ANS"
ANS=$(echo "0: $ANS" | xxd -r -p | openssl enc -aes-256-cbc -d -nopad -nosalt -K "$key_for_ciphering" -iv $IV | xxd -p | tr -d '\n')
echo "Decrypted: $ANS"
---- Edit: ----
Doesn't work, as shell-parameters cannot contain binary zero. Possible fix with filters:
#!/bin/bash
tohex () {
perl -e 'binmode STDIN; while (<STDIN>) { print unpack "H*",$_; }'
}
fromhex () {
perl -e 'binmode STDIN; while (<STDIN>) { print pack "H*",$_; }'
}
binInput='0123456789abcdefghijklmnopqrstuvwxyz'
hexIV="$(openssl rand -hex 16)"
hexKey='788a1ca0bf1ab80f092841aabd77793f'
hexCipher="$(printf '%s' "$binInput" |\
openssl enc -aes-256-cbc -nosalt -iv "$hexIV" -K "$hexKey" | tohex)"
binResult="$(printf '%s' "$hexCipher" | fromhex |\
openssl enc -aes-256-cbc -d -iv "$hexIV" -K "$hexKey")"
if [ "$binInput" = "$binResult" ]; then echo OK;
fi
---- Original: ----
I think your problem lies in hexadecimal conversion. Try using perl pack/unpack:
tohex () {
perl -e 'print unpack "H*", "$ARGV[0]"' "$1"
}
fromhex () {
perl -e 'print pack "H*", "$ARGV[0]"' "$1"
}
message='0123456789abcdefghijklmnopqrstuvwxzy §"+!%/=()'
message_hex=$(tohex "$message")
message_cmp=$(fromhex "$message_hex")
if [ "$message" = "$message_cmp" ]; then echo OK; fi

Generate random passwords in shell with one special character

I have the following code:
</dev/urandom tr -dc 'A-Za-z0-9##$%&_+=' | head -c 16
which is randomly generating passwords perfectly.
I want two changes:
It should only contain one special character listed above
It should choose a random length
I tried with length = $(($RANDOM%8+9))
then putting length as
</dev/urandom tr -dc 'A-Za-z0-9##$%&_+=' | head -c$length
but got no positive result.
#! /bin/bash
chars='##$%&_+='
{ </dev/urandom LC_ALL=C grep -ao '[A-Za-z0-9]' \
| head -n$((RANDOM % 8 + 9))
echo ${chars:$((RANDOM % ${#chars})):1} # Random special char.
} \
| shuf \
| tr -d '\n'
LC_ALL=C prevents characters like ř from appearing.
grep -o outputs just the matching substring, i.e. a single character.
shuf shuffles the lines. I originally used sort -R, but it kept the same characters together (ff1#22MvbcAA).
## Objective: Generating Random Password:
function random_password () {
[[ ${#1} -gt 0 ]] && { local length=${1}; } || { local length=16; }
export DEFAULT_PASSWORDLENGTH=${length};
export LC_CTYPE=C;
local random="$(
tr -cd "[:graph:]" < /dev/urandom \
| head -c ${length} \
| sed -e 's|\`|~|g' \
-e 's|\$(|\\$(|g';
)";
echo -e "${random}";
return 0;
}; alias random-password='random_password';
$ random-password 32 ;
)W#j*deZ2#eMuhU4TODO&eu&r)&.#~3F
# Warning: Do not consider these other options
# date +%s | sha256sum | base64 | head -c 32 | xargs -0;
# Output: MGFjNDlhMTE2ZWJjOTI4OGI4ZTFiZmEz
# dd if=/dev/urandom count=200 bs=1 2>/dev/null \
# | tr -cd "[:graph:]" \
# | cut -c-${length} \
# | xargs -0;
# Output: AuS*D=!wkHR.4DZ_la

script to download file from Amazon S3 bucket

Trying to write script to download file from Amazon S3 bucket.
Having trouble with the example on the cURL site. The script below produces:
The request signature we calculated does not match the signature you
provided. Check your key and signing method.
Appreciate any help.
#!/bin/sh
file="filename.php"
bucket="my-bucket"
resource="/${bucket}/${file}"
contentType="text/html"
dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`"
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key='ABCABCABCABCABCABCAB'
s3Secret='xyzxyzyxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzx'
signature=`/bin/echo -en "$stringToSign" | openssl sha1 -hmac ${s3Secret} - binary | base64`
curl -v -H "Host:lssngen-updates-east.s3.amazonaws.com" \
-H "Date:${dateValue}" \
-H "Content-Type:${contentType}" \
-H "Authorization: AWS ${s3Key}:${signature}" \
https://${bucket}.s3.amazonaws.com/${file}
I write this bash script to download file from s3 (I download compressed file, you can change contentType to download other types of file)
#!/bin/sh
outputFile="Your_PATH"
amzFile="AMAZON_FILE_PATH"
bucket="YOUR_BUCKET"
resource="/${bucket}/${amzFile}"
contentType="application/x-compressed-tar"
dateValue=`date -R`
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key="YOUR_S3_KEY"
s3Secret="YOUR_S3SECRET"
signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -H "Host: ${bucket}.s3.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Content-Type: ${contentType}" \
-H "Authorization: AWS ${s3Key}:${signature}" \
https://${bucket}.s3.amazonaws.com/${amzFile} -o $outputFile
Avoid signing the request yourself, a lot can go wrong or be hard to do. For example, you should check that the date is set to GMT or use x-amz-date headers.
Another approach is to use the AWS Command Line Interface and so use $ aws s3 cp or $ aws s3 sync.
As of August 2019 I found this to work. Has added region, and format of URL has changed.
#!/bin/sh
outputFile="/PATH/TO/LOCALLY/SAVED/FILE"
amzFile="BUCKETPATH/TO/FILE"
region="YOUR-REGION"
bucket="SOME-BUCKET"
resource="/${bucket}/${amzFile}"
contentType="binary/octet-stream"
dateValue=`TZ=GMT date -R`
# You can leave our "TZ=GMT" if your system is already GMT (but don't have to)
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key="ACCESS_KEY_ID"
s3Secret="SECRET_ACCESS_KEY"
signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -H "Host: s3-${region}.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Content-Type: ${contentType}" \
-H "Authorization: AWS ${s3Key}:${signature}" \
https://s3-${region}.amazonaws.com/${bucket}/${amzFile} -o $outputFile
One really effective solution (working on Dec 2021) is to use this script. Just need export keys before use (or copy it's values to the .sh file).
export AWS_ACCESS_KEY_ID=AKxxx
export AWS_SECRET_ACCESS_KEY=zzzz
To download just run
./s3download.sh get s3://mybucket/myfile.txt myfile.txt
All you need to pass get, the s3 bucket along with file name, and output file.
Script needed
Create a s3download.sh file, and chmod +x s3download.sh, to use on command above.
#!/bin/bash
set -eu
s3simple() {
local command="$1"
local url="$2"
local file="${3:--}"
# todo: nice error message if unsupported command?
if [ "${url:0:5}" != "s3://" ]; then
echo "Need an s3 url"
return 1
fi
local path="${url:4}"
if [ -z "${AWS_ACCESS_KEY_ID-}" ]; then
echo "Need AWS_ACCESS_KEY_ID to be set"
return 1
fi
if [ -z "${AWS_SECRET_ACCESS_KEY-}" ]; then
echo "Need AWS_SECRET_ACCESS_KEY to be set"
return 1
fi
local method md5 args
case "$command" in
get)
method="GET"
md5=""
args="-o $file"
;;
put)
method="PUT"
if [ ! -f "$file" ]; then
echo "file not found"
exit 1
fi
md5="$(openssl md5 -binary $file | openssl base64)"
args="-T $file -H Content-MD5:$md5"
;;
*)
echo "Unsupported command"
return 1
esac
local date="$(date -u '+%a, %e %b %Y %H:%M:%S +0000')"
local string_to_sign
printf -v string_to_sign "%s\n%s\n\n%s\n%s" "$method" "$md5" "$date" "$path"
local signature=$(echo -n "$string_to_sign" | openssl sha1 -binary -hmac "${AWS_SECRET_ACCESS_KEY}" | openssl base64)
local authorization="AWS ${AWS_ACCESS_KEY_ID}:${signature}"
curl $args -s -f -H Date:"${date}" -H Authorization:"${authorization}" https://s3.amazonaws.com"${path}"
}
s3simple "$#"
You can find more information about the s3simple script here.
#!/bin/sh
# This works for cross region
outputFile="/PATH/TO/FILE"
awsFile="BUCKETPATH/TO/FILE"
bucket="SOME-BUCKET"
resource="/${bucket}/${awsFile}"
contentType="application/x-compressed-tar"
# Change the content type as desired
dateValue=`TZ=GMT date -R`
#Use dateValue=`date -R` if your TZ is already GMT
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key="ACCESS_KEY_ID"
s3Secret="SECRET_ACCESS_KEY"
signature=`echo -n ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -H "Host: ${bucket}.s3.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Content-Type: ${contentType}" \
-H "Authorization: AWS ${s3Key}:${signature}" \
https://${bucket}.s3.amazonaws.com/${awsFile} -o $outputFile
I created a complete AWS signature creator. You can find the original file here (please note: it has multiple gist comments so it definitely works!).
readonly AWS_ACCESS_KEY_ID='<your_access_key_id>'
readonly AWS_SECRET_ACCESS_KEY='<your_secret_access_key>'
readonly AWS_SERVICE='s3'
readonly AWS_REGION='us-east-1'
readonly AWS_S3_BUCKET_NAME='<your_bucket_name>'
readonly AWS_SERVICE_ENDPOINT_URL="\
${AWS_S3_BUCKET_NAME}.${AWS_SERVICE}.amazonaws.com"
# Create an SHA-256 hash in hexadecimal.
# Usage:
# hash_sha256 <string>
function hash_sha256 {
printf "${1}" | openssl dgst -sha256 | sed 's/^.* //'
}
# Create an SHA-256 hmac in hexadecimal format.
# Usage:
# hmac_sha256 <key> <data>
function hmac_sha256 {
key="$1"
data="$2"
printf "${data}" | openssl dgst -sha256 -mac HMAC -macopt "${key}" | \
sed 's/^.* //'
}
readonly CURRENT_DATE_DAY="$(date -u '+%Y%m%d')"
readonly CURRENT_DATE_TIME="$(date -u '+%H%M%S')"
readonly CURRENT_DATE_ISO8601="${CURRENT_DATE_DAY}T${CURRENT_DATE_TIME}Z"
readonly HTTP_REQUEST_METHOD='GET'
readonly HTTP_REQUEST_PAYLOAD=''
readonly HTTP_REQUEST_PAYLOAD_HASH="$(printf "${HTTP_REQUEST_PAYLOAD}" | \
openssl dgst -sha256 | sed 's/^.* //')"
readonly HTTP_CANONICAL_REQUEST_URI='/video_clips/0940.m3u8'
readonly HTTP_CANONICAL_REQUEST_QUERY_STRING=''
readonly HTTP_REQUEST_CONTENT_TYPE='application/x-www-form-urlencoded'
readonly HTTP_CANONICAL_REQUEST_HEADERS="\
content-type:${HTTP_REQUEST_CONTENT_TYPE}
host:${AWS_SERVICE_ENDPOINT_URL}
x-amz-content-sha256:${HTTP_REQUEST_PAYLOAD_HASH}
x-amz-date:${CURRENT_DATE_ISO8601}"
# Note: The signed headers must match the canonical request headers.
readonly HTTP_REQUEST_SIGNED_HEADERS="\
content-type;host;x-amz-content-sha256;x-amz-date"
readonly HTTP_CANONICAL_REQUEST="\
${HTTP_REQUEST_METHOD}
${HTTP_CANONICAL_REQUEST_URI}
${HTTP_CANONICAL_REQUEST_QUERY_STRING}
${HTTP_CANONICAL_REQUEST_HEADERS}\n
${HTTP_REQUEST_SIGNED_HEADERS}
${HTTP_REQUEST_PAYLOAD_HASH}"
# Create the signature.
# Usage:
# create_signature
function create_signature {
stringToSign="AWS4-HMAC-SHA256
${CURRENT_DATE_ISO8601}
${CURRENT_DATE_DAY}/${AWS_REGION}/${AWS_SERVICE}/aws4_request
$(hash_sha256 "${HTTP_CANONICAL_REQUEST}")"
dateKey=$(hmac_sha256 key:"AWS4${AWS_SECRET_ACCESS_KEY}" \
"${CURRENT_DATE_DAY}")
regionKey=$(hmac_sha256 hexkey:"${dateKey}" "${AWS_REGION}")
serviceKey=$(hmac_sha256 hexkey:"${regionKey}" "${AWS_SERVICE}")
signingKey=$(hmac_sha256 hexkey:"${serviceKey}" "aws4_request")
printf "${stringToSign}" | openssl dgst -sha256 -mac HMAC -macopt \
hexkey:"${signingKey}" | awk '{print $2}'
}
readonly SIGNATURE="$(create_signature)"
readonly HTTP_REQUEST_AUTHORIZATION_HEADER="\
AWS4-HMAC-SHA256 Credential=${AWS_ACCESS_KEY_ID}/${CURRENT_DATE_DAY}/\
${AWS_REGION}/${AWS_SERVICE}/aws4_request, \
SignedHeaders=${HTTP_REQUEST_SIGNED_HEADERS};x-amz-date, Signature=${SIGNATURE}"
curl -X "${HTTP_REQUEST_METHOD}" -v \
"https://${AWS_SERVICE_ENDPOINT_URL}${HTTP_CANONICAL_REQUEST_URI}" \
-H "Authorization: ${HTTP_REQUEST_AUTHORIZATION_HEADER}" \
-H "content-type: ${HTTP_REQUEST_CONTENT_TYPE}" \
-H "x-amz-content-sha256: ${HTTP_REQUEST_PAYLOAD_HASH}" \
-H "x-amz-date: ${CURRENT_DATE_ISO8601}"
Please note though, it's much better to use the AWS APIs if you don't have a reason to create a signature.

Resources