Accessing Azure blob storage using bash, curl - bash

I am attempting to use the Azure blob storage service from a bash script using the REST API. I know it is possible to accomplish this using various other tools or languages, however I'd like to do it as a bash script.
The script below is an attempt to list the blobs in an Azure storage container.
This script results in an authentication error. The signing string and headers look correct based on the REST API (reference) documentation. I suspect the problem may be in juggling the various parts of the signing process.
Has anyone successfully used bash and curl to access cloud storage resources like Azure or other providers?
#!/bin/bash
# List the blobs in an Azure storage container.
echo "usage: ${0##*/} <storage-account-name> <container-name> <access-key>"
storage_account="$1"
container_name="$2"
access_key="$3"
blob_store_url="blob.core.windows.net"
authorization="SharedKey"
request_method="GET"
request_date=$(TZ=GMT date "+%a, %d %h %Y %H:%M:%S %Z")
storage_service_version="2011-08-18"
# HTTP Request headers
x_ms_date_h="x-ms-date:$request_date"
x_ms_version_h="x-ms-version:$storage_service_version"
# Build the signature string
canonicalized_headers="${x_ms_date_h}\n${x_ms_version_h}"
canonicalized_resource="/${storage_account}/${container_name}"
string_to_sign="${request_method}\n\n\n\n\n\n\n\n\n\n\n\n${canonicalized_headers}\n${canonicalized_resource}\ncomp:list\nrestype:container"
# Decode the Base64 encoded access key, convert to Hex.
decoded_hex_key="$(echo -n $access_key | base64 -d -w0 | xxd -p -c256)"
# Create the HMAC signature for the Authorization header
signature=$(echo -n "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" | sed 's/^.*= //' | base64 -w0)
authorization_header="Authorization: $authorization $storage_account:$signature"
curl \
-H "$x_ms_date_h" \
-H "$x_ms_version_h" \
-H "$authorization_header" \
"https://${storage_account}.${blob_store_url}/${container_name}?restype=container&comp=list"
Update - The storage service error and the corresponding signing string that the script generated.
Following is what the storage service returns for the AuthenticationFailed error.
<?xml version="1.0" encoding="utf-8"?>
<Error>
<Code>AuthenticationFailed</Code>
<Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:27e6337e-52f3-4e85-98c7-2fabaacd9ebc
Time:2013-11-21T22:10:11.7029042Z</Message>
<AuthenticationErrorDetail>The MAC signature found in the HTTP request
'OGYxYjk1MTFkYmNkMCgzN2YzODQwNzcyNiIyYTQxZDg0OWFjNGJiZDlmNWY5YzM1ZWQzMWViMGFjYTAyZDY4NAo='
is not the same as any computed signature. Server used following string to sign:
'GET
x-ms-date:Thu, 21 Nov 2013 22:10:11 GMT
x-ms-version:2011-08-18
/storage_account_name/storage_container
comp:list
restype:container'
</AuthenticationErrorDetail>
</Error>
Next is the string_to_sign that the script generates.
GET\n\n\n\n\n\n\n\n\n\n\n\nx-ms-date:Thu, 21 Nov 2013 22:10:11 GMT\nx-ms-version:2011-08-18\n/storage_account_name/storage_container\ncomp:list\nrestype:container

I was able to get it working.
There were two things wrong with this code, the first, as Patrick Park noted, was replacing the echo -n with printf. The second was replacing the sed magic with the -binary option on openssl.
Compare the original:
signature=$(echo -n "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" -binary | sed 's/^.*= //' | base64 -w0)
with the fixed:
signature=$(printf "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" -binary | base64 -w0)
The echo change is needed because echo -n will not convert the \n into actual newlines.
The -binary change is needed because even though you are stripping off the bad part, openssl was still outputting the signature in ascii-encoded-hex, not in binary. So after it was passed to base64, the result was the b64 encoded version of the hex representation, instead of the raw value.

Use Fiddler (or an equivalent on your platform) to intercept the call to Windows Azure Storage. On failure, this will show you the string that the Storage Service used to authenticate the call and you can compare this with the one you used.

Looking at the REST API documentation and your code above, I believe there's an issue with the way you're constructing canonicalized_resource string. You're missing the query parameters in that string. Your canonicalized_resource string should be:
canonicalized_resource="/${storage_account}/${container_name}\ncomp:list\nrestype:container"

It looks like openssl dgst does not generate proper HMAC for you.
I wrote a simple program in C that does the following:
Takes base64-encoded key from the command line and decodes it into binary.
Reads string to sign from standard input.
Uses libcrypto HMAC() routine to generate the signature.
base64-encodes the signature and prints the result to standard output.
I then replaced openssl dgst pipeline in your script with the call to my program and it did the trick.
Please note that the output you are getting from Azure is XML-wrapped and base-64 encoded, so you'll need to come up with some sort of parsing/conversion code for it.

use printf instead of echo (it works for me)
for example:
SIGNATURE=printf "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt hexkey:$HEXKEY -binary | base64 -w0

Related

Sed to replace a script variable by certificate

I'd like to replace a variable in a script template by a public and private certificate.
For example, I've generated a harbor.crt public certificate and a harbor.key private key with the following command:
sudo openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes -keyout /data/harbor.key -out /data/harbor.crt -subj "/CN=$LOCAL_IP" -addext "subjectAltName=IP:127.0.0.1,IP:$LOCAL_IP"
In a template script, I've the following variables I'd like to replace with the above files:
CFG_HARBOR_CRT="CRT" # Harbor registry certificate
CFG_HARBOR_KEY="KEY" # Harbor registry key
To replace those values, I've tried to do something like that:
HARBOR_CRT=`sudo cat /data/harbor.crt`
HARBOR_KEY=`sudo cat /data/harbor.key`
sudo sed -i "s/CFG_HARBOR_CRT\=\"[^\"]*\"/CFG_HARBOR_CRT\=\"$HARBOR_CRT\"/g" ./template-script.sh
sudo sed -i "s/CFG_HARBOR_KEY\=\"[^\"]*\"/CFG_HARBOR_KEY\=\"$HARBOR_KEY\"/g" ./template-script.sh
But both commands failed on: sed: -e expression #1, char 70: unterminated s' command`
Is there a way to use sed command with unescaped variables ?
I suspect there's info missing here. Why use sed at all?
For the simple case, just replace the markers with file reads.
CFG_HARBOR_CRT="$(</data/harbor.crt)"
CFG_HARBOR_KEY="$(</data/harbor.key)"
That might mean you need to run the whole script with elevated priv's though, so I understand why you might not want to do that.
...do you need root to read those files?
If so, and if you don't want the whole script run as root, maybe this:
$: sed 's,^CFG_HARBOR_CRT="CRT",CFG_HARBOR_CRT="$(sudo cat /data/harbor.crt)",
s,^CFG_HARBOR_KEY="KEY",CFG_HARBOR_KEY="$(sudo cat /data/harbor.key)",' tmpf
CFG_HARBOR_CRT="$(sudo cat /data/harbor.crt)" # Harbor registry certificate
CFG_HARBOR_KEY="$(sudo cat /data/harbor.key)" # Harbor registry key
Switching / to , as the demarcation reduces leaning toothpick syndrome.
Switching `...` to $(...) improves flexibility, stability, readability, etc.
Pulling out of comments to get better visibility ...
Consider running the files through base64 and embedding the result into the script, then on the other end run base64 -d to decrypt the data and store in the target files.
Using base64 encoded data should eliminate most (all?) of the sed headaches of dealing with special characters and/or trying to find a sed script delimiter that's not in the data.
OP/Manitoba's reply comment:
That did the trick. I used HARBOR_CRT=$(sudo cat /data/harbor.crt | base64 -w 0) to convert certificate to B64 and echo $CFG_HARBOR_CRT | base64 --decode to decode.

Generating a SHA1 Signature with Different Results Between Ubuntu/Bash and Windows/Powershell

Why would these respective bash and powershell scripts return different signatures:
## (Ubuntu) Bash
now='Fri, 26 Jul 2019 12:32:36 -0400'
bucket="mybucket"
path="/"
awsKey="MSB0M3K06ELMOI65QXI1"
awsSsecret="706Fdj+LFKf8pf/2Wh5V8Q8jbgGUUQo3xSXr5sbt"
## Create the signature
message="GET\n\n\n${now}\n/${bucket}${path}"
messageSignature=$(echo -en "${message}" | openssl sha1 -hmac "${awsSsecret}" -binary | openssl base64)
echo $messageSignature
Returns >> 5oJM2L06LeTcbYSfN3fDFZ9yt5k=
## Powershell
$OutputEncoding = [Text.UTF8Encoding]::UTF8
$now='Fri, 26 Jul 2019 12:32:36 -0400'
$bucket="mybucket"
$path="/"
$awsKey="MSB0M3K06ELMOI65QXI1"
$awsSsecret="706Fdj+LFKf8pf/2Wh5V8Q8jbgGUUQo3xSXr5sbt"
## Create the signature
$message="GET\n\n\n${now}\n/${bucket}${path}"
$messageSignature=$(echo -en "${message}" | openssl sha1 -hmac "${awsSsecret}" -binary | openssl base64)
echo $messageSignature
Returns >> 77u/W8O5RwJeBsOodDddXeKUlCQD4pSUduKVrD3ilIxADQo=
]2
On Ubuntu, my shell is running "en_US.UTF-8".
I've run into the case where the signature is different on different systems: AIX, Windows w/Ubuntu, Windows w/Powershell, etc. I'm trying to figure out why.
There are at least three problems here: Powershell pipelines aren't binary-safe, you have the wrong variable name, and echo isn't portable/reliable.
At least according to this question, you can't pipe binary data in Powershell. The output of openssl (which is raw binary data) is getting treated as UTF-8 text (presumably due to $OutputEncoding), and mangled in the process. You can tell by decoding from base64 and looking in hex instead:
$ echo '77u/W8O5RwJeBsOodDddXeKUlCQD4pSUduKVrD3ilIxADQo=' | base64 -D | xxd
00000000: efbb bf5b c3b9 4702 5e06 c3a8 7437 5d5d ...[..G.^...t7]]
00000010: e294 9424 03e2 9494 76e2 95ac 3de2 948c ...$....v...=...
00000020: 400d 0a #..
It starts with EF BB BF, which is a UTF-8 byte order mark; and it ends with 0D 0A, which is a DOS/Windows line ending (ASCII carriage return and linefeed characters). Something else bad is happening as well, since it's much too long for a sha1 hash, even if you account for the BOM and line ending.
The output of echo is probably getting mangled similarly, so even if the hash wasn't mangled it'd be the hash of the wrong byte sequence.
See this question (and its answer) for an example of using Powershell's own tools to compute the HMAC-SHA1 of a string.
The message to be signed is in $message, but you actually sign $string_to_sign, which is undefined. The Ubuntu result is the correct HMAC-SHA1 for the null string:
$ </dev/null openssl sha1 -hmac "706Fdj+LFKf8pf/2Wh5V8Q8jbgGUUQo3xSXr5sbt" -binary | openssl base64
NCRRWG4nL9sN8QMrdmCPmUvNlYA=
As Lorinczy Zsigmond pointed out, echo -en isn't predictable. Under some implementations/OSes/shells/compile & runtime options/phases of moon/etc, it might or might not print "-en" as part of its output, and maybe also print a newline (or something) at the end. Use printf instead:
printf 'GET\n\n\n%s\n/%s%s' "${now}" "${bucket}" "${path}" | openssl sha1 ...
Or (in bash, but not all other shells):
printf -v message 'GET\n\n\n%s\n/%s%s' "${now}" "${bucket}" "${path}"
printf '%s' "$message" | openssl sha1 ...
Or (bash again):
nl=$'\n'
message="GET${nl}${nl}${nl}${now}${nl}/${bucket}${path}"
printf '%s' "$message" | openssl sha1 ...
Providing a summary for anyone who stumbled onto this thread. The problem problems were....
Cannot pass binary data through the pipeline as noted by Gordon Davisson
Need to represent newlines with "`n" instead of "\n"
The Powershell script that ultimately replicated my results in Ubuntu/Bash was...
$now='Fri, 26 Jul 2019 12:32:36 -0400'
$bucket="mybucket"
$path="/"
$awsKey="MSB0M3K06ELMOI65QXI1"
$awsSsecret="706Fdj+LFKf8pf/2Wh5V8Q8jbgGUUQo3xSXr5sbt"
$message="GET`n`n`n${now}`n/${bucket}${path}"
## Create the signature
$sha = [System.Security.Cryptography.KeyedHashAlgorithm]::Create("HMACSHA1")
$sha.Key = [System.Text.Encoding]::UTF8.Getbytes($awsSsecret)
[Convert]::Tobase64String($sha.ComputeHash([System.Text.Encoding]::UTF8.Getbytes($message)))
While it's a better idea to use printf over echo on Linux/Unix systems, that difference wasn't material in this use-case.

How to use OpenSSL to generate SHARSA1 base64 signature

I have generated a public/private keypair with OpenSSL. I want to use the private key now to sign my message using OpenSSL, and I was thinking to stay in a bash environment. I am required to use SHA-RSA1.
So far, I was suggested the following code but I am not happy with it:
openssl.exe dgst -sha1 -sign C:\...\path\to\key\privatekey.pem -binary C:\...\path\to\message\message.txt
I don't want to have my message be stored in a file (message.txt) to generate a signature and in any case, I would need to use openssl base64 afterwards to get the base64 representation.
Is there a more proper way to achieve what I want (and a one liner would be great)?
Use openssl itself to encode base64
echo "$msg" | openssl dgst ... -binary | openssl enc -base64

How to calculate sha1 base64 encoded in windows batch?

I am trying to get a base64 encoded sha1 hash in a windows batch file.
The first thing I tried was with perl:
perl -M"Digest::SHA1 qw(sha1_base64)" -e "open(F,shift) or die; binmode F; print sha1_base64(<F>), qq(=\n)" "test.mxf"
This works great, but only for small files. With big files it says "Out of memory".
Then I downloaded an openssl version for windows and tried this:
"C:\openssl.exe" dgst -sha1 -binary -out "hash_sha1.txt" "C:\test.mxf"
set /p hash_sha1=<"hash_sha1.txt"
del "hash_sha1.txt"
echo !hash_sha1!
echo -n '!hash_sha1!' | "C:\openssl.exe" enc -base64
But the output of the openssl method is different from the Perl output and I know that the Perl method produces the correct output. What do I have to change?
There's no -n parameter of echo so -n AND single quotes are part of the output.
The intermediate files and variables aren't needed, use piping.
The entire code:
openssl dgst -sha1 -binary "C:\test.mxf" | openssl enc -base64
If you create a Digest::SHA1 object, you can use the add method to calculate the hash incrementally
There is also no need to explicitly open files passed as command-line parameters. They are opened automatically using the built-in file handle ARGV, and can be read with the empoty diamond operator <>
perl -Mopen=IN,:raw -MDigest::SHA1 -e"$d=Digest::SHA1->new; $d->add($_) while <>; print $d->b64digest, qq{=\n}" 5GB.bin
This command line was quite happy to generate the SHA1 hash of a 5GB file, but if you are unlucky enough to have a very big file that contains no linefeeds then you will have to set a read block size with something like
local $/ = \(1024*1024)

Encrypting using 64-bit DES with openssl

I'm trying to do a very simple thing, namely using a 64-bit password and a 64-bit plaintext (both in hex) and encrypt it with simple old DES.
my script looks like this:
plaintext=`echo -n "$2" | sed 's/\(..\)/\\\x\1/g'`
key=$1
printf "$plaintext" | openssl enc -nosalt -e -des -nopad -K "$key" -iv "0000000000000000" | od --format=x1 --width=32 --address-radix=n | sed 's/ //g'
I execute and get the following result:
./des_enc 5B5A57676A56676E 675A69675E5A6B5A
b617e2c84a4fba2149dd7132433031392257b99d9284b1031c351c15825aca52
The problem is there's too much data coming back from openssl, I expect to only get 64-bits of data instead I get 512. I don't know how to explicit request a 64-bit version of DES, is it even possible?
Note: The values used above are from "H. Katzan, The Standard Data Encryption Algorithm, pp75-94, Petrocelli Books Inc., New York, 1977" is:
Key: 5B5A57676A56676E
Plaintext: 675A69675E5A6B5A
Ciphertext: 974AFFBF86022D1F
Use -des-ecb. Also, xxd makes this pipeline much cleaner, if you have it handy (it's part of the vim package):
sh % echo 675A69675E5A6B5A | xxd -r -ps | openssl enc -des-ecb -nopad -K 5B5A57676A56676E | xxd -ps
974affbf86022d1f

Resources