I am trying to get a base64 encoded sha1 hash in a windows batch file.
The first thing I tried was with perl:
perl -M"Digest::SHA1 qw(sha1_base64)" -e "open(F,shift) or die; binmode F; print sha1_base64(<F>), qq(=\n)" "test.mxf"
This works great, but only for small files. With big files it says "Out of memory".
Then I downloaded an openssl version for windows and tried this:
"C:\openssl.exe" dgst -sha1 -binary -out "hash_sha1.txt" "C:\test.mxf"
set /p hash_sha1=<"hash_sha1.txt"
del "hash_sha1.txt"
echo !hash_sha1!
echo -n '!hash_sha1!' | "C:\openssl.exe" enc -base64
But the output of the openssl method is different from the Perl output and I know that the Perl method produces the correct output. What do I have to change?
There's no -n parameter of echo so -n AND single quotes are part of the output.
The intermediate files and variables aren't needed, use piping.
The entire code:
openssl dgst -sha1 -binary "C:\test.mxf" | openssl enc -base64
If you create a Digest::SHA1 object, you can use the add method to calculate the hash incrementally
There is also no need to explicitly open files passed as command-line parameters. They are opened automatically using the built-in file handle ARGV, and can be read with the empoty diamond operator <>
perl -Mopen=IN,:raw -MDigest::SHA1 -e"$d=Digest::SHA1->new; $d->add($_) while <>; print $d->b64digest, qq{=\n}" 5GB.bin
This command line was quite happy to generate the SHA1 hash of a 5GB file, but if you are unlucky enough to have a very big file that contains no linefeeds then you will have to set a read block size with something like
local $/ = \(1024*1024)
Related
I'd like to replace a variable in a script template by a public and private certificate.
For example, I've generated a harbor.crt public certificate and a harbor.key private key with the following command:
sudo openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes -keyout /data/harbor.key -out /data/harbor.crt -subj "/CN=$LOCAL_IP" -addext "subjectAltName=IP:127.0.0.1,IP:$LOCAL_IP"
In a template script, I've the following variables I'd like to replace with the above files:
CFG_HARBOR_CRT="CRT" # Harbor registry certificate
CFG_HARBOR_KEY="KEY" # Harbor registry key
To replace those values, I've tried to do something like that:
HARBOR_CRT=`sudo cat /data/harbor.crt`
HARBOR_KEY=`sudo cat /data/harbor.key`
sudo sed -i "s/CFG_HARBOR_CRT\=\"[^\"]*\"/CFG_HARBOR_CRT\=\"$HARBOR_CRT\"/g" ./template-script.sh
sudo sed -i "s/CFG_HARBOR_KEY\=\"[^\"]*\"/CFG_HARBOR_KEY\=\"$HARBOR_KEY\"/g" ./template-script.sh
But both commands failed on: sed: -e expression #1, char 70: unterminated s' command`
Is there a way to use sed command with unescaped variables ?
I suspect there's info missing here. Why use sed at all?
For the simple case, just replace the markers with file reads.
CFG_HARBOR_CRT="$(</data/harbor.crt)"
CFG_HARBOR_KEY="$(</data/harbor.key)"
That might mean you need to run the whole script with elevated priv's though, so I understand why you might not want to do that.
...do you need root to read those files?
If so, and if you don't want the whole script run as root, maybe this:
$: sed 's,^CFG_HARBOR_CRT="CRT",CFG_HARBOR_CRT="$(sudo cat /data/harbor.crt)",
s,^CFG_HARBOR_KEY="KEY",CFG_HARBOR_KEY="$(sudo cat /data/harbor.key)",' tmpf
CFG_HARBOR_CRT="$(sudo cat /data/harbor.crt)" # Harbor registry certificate
CFG_HARBOR_KEY="$(sudo cat /data/harbor.key)" # Harbor registry key
Switching / to , as the demarcation reduces leaning toothpick syndrome.
Switching `...` to $(...) improves flexibility, stability, readability, etc.
Pulling out of comments to get better visibility ...
Consider running the files through base64 and embedding the result into the script, then on the other end run base64 -d to decrypt the data and store in the target files.
Using base64 encoded data should eliminate most (all?) of the sed headaches of dealing with special characters and/or trying to find a sed script delimiter that's not in the data.
OP/Manitoba's reply comment:
That did the trick. I used HARBOR_CRT=$(sudo cat /data/harbor.crt | base64 -w 0) to convert certificate to B64 and echo $CFG_HARBOR_CRT | base64 --decode to decode.
What would be the easiest way to convert the text produced by utilities such as sha512sum into a binary file?
I'd like to convert hex string like 77f4de214a5423e3c7df8704f65af57c39c55f08424c75c0931ab09a5bfdf49f5f939f2caeff1e0113d9b3d6363583e4830cf40787100b750cde76f00b8cd3ec (example produced by sha512sum) into a binary file (64 bytes long), in which each byte's value would be equivalent to a pair of letters/digits from the string. I'm looking for a solution that would require minimal amount of tools, so I'd be happy if this can be done easily with bash, sed or some utility from coreutils. I'd rather avoid xxd, as this doesn't seem to handle such string anyway (I'd have to add "addresses" and some whitespace).
I need the hash as a binary file, to convert it into an object file and link with the application that I'm writing. If there's another way to embed such string (in a binary form!) into application (via an array or whatever) - it's also a solution for me.
A bit of sed and echo might do:
for i in $(echo 77f4de214a5423e3c7df8704f65af57c39c55f08424c75c0931ab09a5bfdf49f5f939f2caeff1e0113d9b3d6363583e4830cf40787100b750cde76f00b8cd3ec | sed 's/../& /g'); do
echo -ne "\x$i"
done > output.bin
The sed command is splitting the hex string into bytes and the echo shows it as hexadecimal character.
Or in a shorter form with sha512sum output, as suggested in the comment:
echo -ne "$(sha512sum some-file.txt | sed 's/ .*$//; s/../\\x&/g')"
How about perl:
<<<77f4de214a5423e3c7df8704f65af57c39c55f08424c75c0931ab09a5bfdf49f5f939f2caeff1e0113d9b3d6363583e4830cf40787100b750cde76f00b8cd3ec \
perl -e 'print pack "H*", <STDIN>' > hash.bin
If you have openssl in your system and want a sha512 hash in binary form, you can use this:
openssl dgst -sha512 -binary somefile.txt
If you have node:
node -e "var fs = require('fs'); fs.writeFileSync('binary', new Buffer('77f4de214a5423e3c7df8704f65af57c39c55f08424c75c0931ab09a5bfdf49f5f939f2caeff1e0113d9b3d6363583e4830cf40787100b750cde76f00b8cd3ec', 'hex'))"
s="77f4de214a5423e3c7df8704f65af57c39c55f08424c75c0931ab09a5bfdf49f5f939f2caeff1e0113d9b3d6363583e4830cf40787100b750cde76f00b8cd3ec";
echo -n $s | xxd -r -p > file.bin
1 File(s) 64 bytes
Tested on Ubuntu 16.04.7
I am trying to calculate the digests of a file using md5 algorithm. I am asked to format the output to be in binary not hex. So this is my command in terminal (I use mac):
openssl dgst -md5 -binary ~/Documents/msg_Joudi.txt > ~/Documents/hash.txt
this generates hash.txt file, but its content is not in binary format, and I do not know where is the error.
Create MD5 hash of file: msgFile.txt > Convert to Binary and save:
cat msgFile.txt | openssl dgst -md5 -binary > hash.bin.txt
Save binary in Base64 format:
cat msgFile.txt | openssl dgst -md5 -binary | base64 > hash.bin.b64.txt
Save binary in Hexadecimal representation:
cat msgFile.txt | openssl dgst -md5 -binary | xxd -b > hash.bin.hex.txt
The OP says he wants binary and then says "Binary should be 0s and 1s right?".
Although, probably unlikely, if he really wants a binary number output, do this:
$ echo -e "xx\nyy" >in.txt
$ perl -ln0777e 'use Digest::MD5 "md5"; print "0b".unpack("B*",md5($_))' <in.txt
0b001111000101011000010000100101100011010101000101111000101000000011110110011010110011010000!
Meaning of above:
slurp in whole file into $_, calculate md5 digest on $_ to create binary data
use unpack to convert binary data to binary number string
If he really wants binary data, modify the above:
$ perl -ln0777e 'use Digest::MD5 "md5"; print md5 $_' <in.txt >binout.txt
This problem is really leaving me astounded
Take a input file and create a HMAC_256 value from it using a private key
Base64 encode HMAC_256 hash
Code
#Create HMAC-SHA2 hash from shell parameter
filehash=`echo $1 | mac -a sha256_hmac -k test.key`
echo "HMAC_SHA256 hash : "$filehash
#Base64 encode filehash using openssl
filehash_64=`echo "$filehash" | /usr/sfw/bin//openssl enc -base64 | tr '\n' ' ' | cut -d " " -f2 `
echo "64 bit encoded hash : "$filehash_64
Using a test.key of
Bob123
Shell Input
Hello
Shell Output
SHA256 hash : 411796cfb1e6c30c1b39b589c79d6f8bf1fdde8d58fda4a6ec1e59538ecaa39a
64 bit encoded hash : ZWMxZTU5NTM4ZWNhYTM5YQo=
However if I go to these sites and do a HMAC_256 test they both generate a different hash
http://asecuritysite.com/encryption/hmac
http://jetcityorange.com/hmac/
They both Output a HMAC_256 hash of
a30410f584726f32ba3e6e823bfdecbdf28448d64e4ab8f11f6a2e66818b50fe
Why are they generating a different hash? I am assuming they are correct as they both have the same.
Does Solaris 10 have a bug with its MAC (Message Authentication Code)
tool?
Is it UTF8 or ASCII problem?
Is it a server problem, Windows / Unix?
I don't understand why I am generating a different hash to them, even though I am using the same hashing algorithm and key.
I'll guess that the problem is newline characters. The echo command puts a newline after "Hello", so if you don't want it, use "echo -n". Also make sure that there is no newline character in your key file.
I am attempting to use the Azure blob storage service from a bash script using the REST API. I know it is possible to accomplish this using various other tools or languages, however I'd like to do it as a bash script.
The script below is an attempt to list the blobs in an Azure storage container.
This script results in an authentication error. The signing string and headers look correct based on the REST API (reference) documentation. I suspect the problem may be in juggling the various parts of the signing process.
Has anyone successfully used bash and curl to access cloud storage resources like Azure or other providers?
#!/bin/bash
# List the blobs in an Azure storage container.
echo "usage: ${0##*/} <storage-account-name> <container-name> <access-key>"
storage_account="$1"
container_name="$2"
access_key="$3"
blob_store_url="blob.core.windows.net"
authorization="SharedKey"
request_method="GET"
request_date=$(TZ=GMT date "+%a, %d %h %Y %H:%M:%S %Z")
storage_service_version="2011-08-18"
# HTTP Request headers
x_ms_date_h="x-ms-date:$request_date"
x_ms_version_h="x-ms-version:$storage_service_version"
# Build the signature string
canonicalized_headers="${x_ms_date_h}\n${x_ms_version_h}"
canonicalized_resource="/${storage_account}/${container_name}"
string_to_sign="${request_method}\n\n\n\n\n\n\n\n\n\n\n\n${canonicalized_headers}\n${canonicalized_resource}\ncomp:list\nrestype:container"
# Decode the Base64 encoded access key, convert to Hex.
decoded_hex_key="$(echo -n $access_key | base64 -d -w0 | xxd -p -c256)"
# Create the HMAC signature for the Authorization header
signature=$(echo -n "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" | sed 's/^.*= //' | base64 -w0)
authorization_header="Authorization: $authorization $storage_account:$signature"
curl \
-H "$x_ms_date_h" \
-H "$x_ms_version_h" \
-H "$authorization_header" \
"https://${storage_account}.${blob_store_url}/${container_name}?restype=container&comp=list"
Update - The storage service error and the corresponding signing string that the script generated.
Following is what the storage service returns for the AuthenticationFailed error.
<?xml version="1.0" encoding="utf-8"?>
<Error>
<Code>AuthenticationFailed</Code>
<Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:27e6337e-52f3-4e85-98c7-2fabaacd9ebc
Time:2013-11-21T22:10:11.7029042Z</Message>
<AuthenticationErrorDetail>The MAC signature found in the HTTP request
'OGYxYjk1MTFkYmNkMCgzN2YzODQwNzcyNiIyYTQxZDg0OWFjNGJiZDlmNWY5YzM1ZWQzMWViMGFjYTAyZDY4NAo='
is not the same as any computed signature. Server used following string to sign:
'GET
x-ms-date:Thu, 21 Nov 2013 22:10:11 GMT
x-ms-version:2011-08-18
/storage_account_name/storage_container
comp:list
restype:container'
</AuthenticationErrorDetail>
</Error>
Next is the string_to_sign that the script generates.
GET\n\n\n\n\n\n\n\n\n\n\n\nx-ms-date:Thu, 21 Nov 2013 22:10:11 GMT\nx-ms-version:2011-08-18\n/storage_account_name/storage_container\ncomp:list\nrestype:container
I was able to get it working.
There were two things wrong with this code, the first, as Patrick Park noted, was replacing the echo -n with printf. The second was replacing the sed magic with the -binary option on openssl.
Compare the original:
signature=$(echo -n "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" -binary | sed 's/^.*= //' | base64 -w0)
with the fixed:
signature=$(printf "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" -binary | base64 -w0)
The echo change is needed because echo -n will not convert the \n into actual newlines.
The -binary change is needed because even though you are stripping off the bad part, openssl was still outputting the signature in ascii-encoded-hex, not in binary. So after it was passed to base64, the result was the b64 encoded version of the hex representation, instead of the raw value.
Use Fiddler (or an equivalent on your platform) to intercept the call to Windows Azure Storage. On failure, this will show you the string that the Storage Service used to authenticate the call and you can compare this with the one you used.
Looking at the REST API documentation and your code above, I believe there's an issue with the way you're constructing canonicalized_resource string. You're missing the query parameters in that string. Your canonicalized_resource string should be:
canonicalized_resource="/${storage_account}/${container_name}\ncomp:list\nrestype:container"
It looks like openssl dgst does not generate proper HMAC for you.
I wrote a simple program in C that does the following:
Takes base64-encoded key from the command line and decodes it into binary.
Reads string to sign from standard input.
Uses libcrypto HMAC() routine to generate the signature.
base64-encodes the signature and prints the result to standard output.
I then replaced openssl dgst pipeline in your script with the call to my program and it did the trick.
Please note that the output you are getting from Azure is XML-wrapped and base-64 encoded, so you'll need to come up with some sort of parsing/conversion code for it.
use printf instead of echo (it works for me)
for example:
SIGNATURE=printf "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt hexkey:$HEXKEY -binary | base64 -w0