This problem is really leaving me astounded
Take a input file and create a HMAC_256 value from it using a private key
Base64 encode HMAC_256 hash
Code
#Create HMAC-SHA2 hash from shell parameter
filehash=`echo $1 | mac -a sha256_hmac -k test.key`
echo "HMAC_SHA256 hash : "$filehash
#Base64 encode filehash using openssl
filehash_64=`echo "$filehash" | /usr/sfw/bin//openssl enc -base64 | tr '\n' ' ' | cut -d " " -f2 `
echo "64 bit encoded hash : "$filehash_64
Using a test.key of
Bob123
Shell Input
Hello
Shell Output
SHA256 hash : 411796cfb1e6c30c1b39b589c79d6f8bf1fdde8d58fda4a6ec1e59538ecaa39a
64 bit encoded hash : ZWMxZTU5NTM4ZWNhYTM5YQo=
However if I go to these sites and do a HMAC_256 test they both generate a different hash
http://asecuritysite.com/encryption/hmac
http://jetcityorange.com/hmac/
They both Output a HMAC_256 hash of
a30410f584726f32ba3e6e823bfdecbdf28448d64e4ab8f11f6a2e66818b50fe
Why are they generating a different hash? I am assuming they are correct as they both have the same.
Does Solaris 10 have a bug with its MAC (Message Authentication Code)
tool?
Is it UTF8 or ASCII problem?
Is it a server problem, Windows / Unix?
I don't understand why I am generating a different hash to them, even though I am using the same hashing algorithm and key.
I'll guess that the problem is newline characters. The echo command puts a newline after "Hello", so if you don't want it, use "echo -n". Also make sure that there is no newline character in your key file.
Related
The following steps are to be performed:
Do base64 decode on ASCII value.
To 'chirp' append the decoded value.
Generate sha256 on the "chirp<decoded_value>"
#!/bin/sh
a=$(echo MDkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDIgADAoR0WUZBTAkZv0Syvt+g5wGpb/HYHh22zAxCNP+ryTQ=|base64 -d)
b="chirp$a"
echo $b
echo -n $b | sha256sum
I am getting a value of:
f62e19108cfb5a91434f1bba9f5384f9039857743aa2c0707efaa092791e4420
But the expected value is:
6a29cb4....
Am I missing anything?
For binary data, as the base64-decoded data that your dealing with, I would not rely too much on echo, but just pipe the stuff, like this:
<<<'MDkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDIgADAoR0WUZBTAkZv0Syvt+g5wGpb/HYHh22zAxCNP+ryTQ=' base64 -d | cat <(echo -n chirp) - | sha256sum
That gives me the result that you expect, 6a29cb438954e8c78241d786af874b1c7218490d3024345f6e11932377a932b6 .
Here, cat gets two file descriptors as arguments, the first one streaming the word "chirp" and the second one forwarding the stdout of the previous command (base64 -d)
I am trying to make a bash script work on MinGW and it seems the shell is not able to decode something like the following.
t=$(openssl rand -base64 64)
echo "$t" | base64 --decode
Resulting in,
Ԋ7▒%
▒7▒SUfX▒L:o<x▒▒䈏ţ94V▒▒▒▒uW;▒▒pxu▒base64: invalid input
Interestingly, if I output the base64 character and run the command as such, it works.
echo "+e5dcWsijZ83uR2cuKxIDJTTiwTvqB7J0EJ63paJdzGomQxw9BhfPvFTkdKP9I1o
g29pZKjUfDO8/SUNt+idWQ==" | base64 --decode
Anybody knows what I am doing wrong?
Thanks
I solved the above case by passing --ignore-garbage flag to the base64 decode. It ignores the non-alphabet characters.
echo "$t" | base64 --decode --ignore-garbage
However, I would still like to know how did I create "garbage" ;) ?
I think what has happened here is the base64 string contains some embedded spaces, and that causes the actual "invalid input" w (and what you observe to as garbage.)
The openssl rand -base64 64 command introduces some newlines (not spaces), for example,
openssl rand -base64 64 > b64.txt
... then viewing the b64.txt file in an editor I see two separate lines
tPKqKPbH5LkGu13KR6zDdJOBpUGD4pAqS6wKGS32EOyJaK0AmTG4da3fDuOI4T+k
abInqlQcH5k7k9ZVEzv8FA==
... and this implies there is a newline character between the 'k' and 'a'
So the base64 string has this embedded newline. The base64 -d can handle newlines (as demonstrated by your successful example), but, it cannot handle space char.
The newlines can get translated to spaces by some actions of the shell. This is very likely happening by the echo $t I.e. if t has newlines inside it, then echo will just replace then with single space. Actually, how it behaves can depend on shell options, and type of string quotes, if any, applied.
To fix the command, we can remove the newline before piping to the base64 -d command.
One way to do that is to use tr command, e.g. the following works on Linux:
t=$(openssl rand -base64 64 | tr -d '\n')
echo $t | base64 -d
... or alternatively, remove the spaces, again using tr
t=$(openssl rand -base64 64)
echo $t | tr -d ' ' | base64 -d
What would be the easiest way to convert the text produced by utilities such as sha512sum into a binary file?
I'd like to convert hex string like 77f4de214a5423e3c7df8704f65af57c39c55f08424c75c0931ab09a5bfdf49f5f939f2caeff1e0113d9b3d6363583e4830cf40787100b750cde76f00b8cd3ec (example produced by sha512sum) into a binary file (64 bytes long), in which each byte's value would be equivalent to a pair of letters/digits from the string. I'm looking for a solution that would require minimal amount of tools, so I'd be happy if this can be done easily with bash, sed or some utility from coreutils. I'd rather avoid xxd, as this doesn't seem to handle such string anyway (I'd have to add "addresses" and some whitespace).
I need the hash as a binary file, to convert it into an object file and link with the application that I'm writing. If there's another way to embed such string (in a binary form!) into application (via an array or whatever) - it's also a solution for me.
A bit of sed and echo might do:
for i in $(echo 77f4de214a5423e3c7df8704f65af57c39c55f08424c75c0931ab09a5bfdf49f5f939f2caeff1e0113d9b3d6363583e4830cf40787100b750cde76f00b8cd3ec | sed 's/../& /g'); do
echo -ne "\x$i"
done > output.bin
The sed command is splitting the hex string into bytes and the echo shows it as hexadecimal character.
Or in a shorter form with sha512sum output, as suggested in the comment:
echo -ne "$(sha512sum some-file.txt | sed 's/ .*$//; s/../\\x&/g')"
How about perl:
<<<77f4de214a5423e3c7df8704f65af57c39c55f08424c75c0931ab09a5bfdf49f5f939f2caeff1e0113d9b3d6363583e4830cf40787100b750cde76f00b8cd3ec \
perl -e 'print pack "H*", <STDIN>' > hash.bin
If you have openssl in your system and want a sha512 hash in binary form, you can use this:
openssl dgst -sha512 -binary somefile.txt
If you have node:
node -e "var fs = require('fs'); fs.writeFileSync('binary', new Buffer('77f4de214a5423e3c7df8704f65af57c39c55f08424c75c0931ab09a5bfdf49f5f939f2caeff1e0113d9b3d6363583e4830cf40787100b750cde76f00b8cd3ec', 'hex'))"
s="77f4de214a5423e3c7df8704f65af57c39c55f08424c75c0931ab09a5bfdf49f5f939f2caeff1e0113d9b3d6363583e4830cf40787100b750cde76f00b8cd3ec";
echo -n $s | xxd -r -p > file.bin
1 File(s) 64 bytes
Tested on Ubuntu 16.04.7
This question already has answers here:
Base 64 encoding from command line gives different output than other methods
(2 answers)
Closed 7 days ago.
Can anyone explain this?
[vagrant#centos ~]$ echo "10IXydrdsc4DVAgxzrXldNw5GMeVAHKG:TAO04JuWz4PBVWYm" | base64
MTBJWHlkcmRzYzREVkFneHpyWGxkTnc1R01lVkFIS0c6VEFPMDRKdVd6NFBCVldZbQo=
[vagrant#centos ~]$ echo "MTBJWHlkcmRzYzREVkFneHpyWGxkTnc1R01lVkFIS0c6VEFPMDRKdVd6NFBCVldZbQ==" | base64 -d
10IXydrdsc4DVAgxzrXldNw5GMeVAHKG:TAO04JuWz4PBVWYm
The first string encodes with o= at the end, but the encoded string with == at the end instead, decodes to the same original string...
GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)
Compare these
echo "10IXydrdsc4DVAgxzrXldNw5GMeVAHKG:TAO04JuWz4PBVWYm" | base64 | od -c
echo "MTBJWHlkcmRzYzREVkFneHpyWGxkTnc1R01lVkFIS0c6VEFPMDRKdVd6NFBCVldZbQ==" | base64 -D | od -c
echo "MTBJWHlkcmRzYzREVkFneHpyWGxkTnc1R01lVkFIS0c6VEFPMDRKdVd6NFBCVldZbQo=" | base64 -D | od -c
If we don't send the newline when using echo the o is missing, have a look at this...
echo -n "10IXydrdsc4DVAgxzrXldNw5GMeVAHKG:TAO04JuWz4PBVWYm" | base64
It's the newline that's being encoded that gives the o in o=
The = is padding and it might not always be there. Have a look here..
https://en.wikipedia.org/wiki/Base64#Padding
Different implementations may also use different padding characters. You can see some of the differences here
https://en.wikipedia.org/wiki/Base64#Variants_summary_table
From the RFC
3.2. Padding of Encoded Data
In some circumstances, the use of padding ("=") in base-encoded
data is not required or used. In the general case, when
assumptions about the size of transported data cannot be made,
padding is required to yield correct decoded data.
Implementations MUST include appropriate pad characters at the end
of encoded data unless the specification referring to this document
explicitly states otherwise.
The base64 and base32 alphabets use padding, as described below in
sections 4 and 6, but the base16 alphabet does not need it; see
section 8.
When you use $echo, a newline is appended to the end of the output. This newline character is part of the base64 encoding. When you change the 'o' to a '=', you're changing the encoding of the newline character. In this case, the character it decodes to is still not a printable character.
In my terminal, decoding the two string yields the same output, but the string ending in "o=" has a newline, and the string ending in "==" does not.
$> echo "MTBJWHlkcmRzYzREVkFneHpyWGxkTnc1R01lVkFIS0c6VEFPMDRKdVd6NFBCVldZbQo=" | base64 -d
10IXydrdsc4DVAgxzrXldNw5GMeVAHKG:TAO04JuWz4PBVWYm
$> echo "MTBJWHlkcmRzYzREVkFneHpyWGxkTnc1R01lVkFIS0c6VEFPMDRKdVd6NFBCVldZbQ==" | base64 -d
10IXydrdsc4DVAgxzrXldNw5GMeVAHKG:TAO04JuWz4PBVWYm $>
Using $echo -n would allow you to pass the string into base64 without the trailing newline. The string without the newline encodes to the string ending in "==".
On Macs, I noticed I had to append "\c" to the end of my string to get it to work, like this:
[vagrant#centos ~]$ echo "10IXydrdsc4DVAgxzrXldNw5GMeVAHKG:TAO04JuWz4PBVWYm\c" | base64
The result was:
MTBJWHlkcmRzYzREVkFneHpyWGxkTnc1R01lVkFIS0c6VEFPMDRKdVd6NFBCVldZbVxjCg==
PHP also encodes it properly, which leads me to believe there is some issue with the base64 program in bash, as I haven't found any mention of 'o' somehow being used as a padding character.
php > echo base64_encode("10IXydrdsc4DVAgxzrXldNw5GMeVAHKG:TAO04JuWz4PBVWYm");
MTBJWHlkcmRzYzREVkFneHpyWGxkTnc1R01lVkFIS0c6VEFPMDRKdVd6NFBCVldZbQ==
I am trying to get a base64 encoded sha1 hash in a windows batch file.
The first thing I tried was with perl:
perl -M"Digest::SHA1 qw(sha1_base64)" -e "open(F,shift) or die; binmode F; print sha1_base64(<F>), qq(=\n)" "test.mxf"
This works great, but only for small files. With big files it says "Out of memory".
Then I downloaded an openssl version for windows and tried this:
"C:\openssl.exe" dgst -sha1 -binary -out "hash_sha1.txt" "C:\test.mxf"
set /p hash_sha1=<"hash_sha1.txt"
del "hash_sha1.txt"
echo !hash_sha1!
echo -n '!hash_sha1!' | "C:\openssl.exe" enc -base64
But the output of the openssl method is different from the Perl output and I know that the Perl method produces the correct output. What do I have to change?
There's no -n parameter of echo so -n AND single quotes are part of the output.
The intermediate files and variables aren't needed, use piping.
The entire code:
openssl dgst -sha1 -binary "C:\test.mxf" | openssl enc -base64
If you create a Digest::SHA1 object, you can use the add method to calculate the hash incrementally
There is also no need to explicitly open files passed as command-line parameters. They are opened automatically using the built-in file handle ARGV, and can be read with the empoty diamond operator <>
perl -Mopen=IN,:raw -MDigest::SHA1 -e"$d=Digest::SHA1->new; $d->add($_) while <>; print $d->b64digest, qq{=\n}" 5GB.bin
This command line was quite happy to generate the SHA1 hash of a 5GB file, but if you are unlucky enough to have a very big file that contains no linefeeds then you will have to set a read block size with something like
local $/ = \(1024*1024)