I'm trying to figure out the equivalent shell script of a small ruby script I have. This is the ruby script:
require 'openssl'
require 'base64'
k = OpenSSL::PKey::RSA.new(File.read("key.pem"))
res = File.read("res.tmp")
digest = OpenSSL::Digest::SHA256.new
signature = k.sign(digest,res)
File.write("foo1.txt",Base64.strict_encode64(signature))
That's it. It takes some data, gets the SHA256 hash of it, and then signs that with a private key I have. I think that the equivalent command on the terminal should be:
openssl sha -sha256 -sign key.pem < res.tmp | base64 -w 0 > foo2.txt
But these do not produce the same output. Could anyone enlighten me as to why?
-- EDIT --
I'm adding some more information so that people can try to reproduce this on their computers if so desired. The contents of res.tmp are:
This is some sample text.
This is some sample text.
This is some sample text.
This is some more sample text.
This is some more sample text.
This is some more sample text.
The private key (which is not a private key that is actually used for any production system, just clarifying that) is:
-----BEGIN PRIVATE KEY-----
MIICdgIBADANBgkqhkiG9w0BAQEFAASCAmAwggJcAgEAAoGBALNY2EtJTj78tvnL
G1v+5HFqn8mZ85AToUqj/n86/eymcMqEjskHAoVmU9FOr+ZsnSxopNvUVLCzDkv6
w88tUtHGblzFtzJpQrxvtH8VCd2BNUc6Esxx7lmQeCxJMMPnmY2ZkDZuH6L+UsVL
DbM7LCvJyluo/A3AP68G9YL+XTjFAgMBAAECgYBkJzsyX894UTwFJq0ypJcB1x9A
P97KGIw72HTorBLdMt1N2tS54lZAFLK98gk8zm6/O/jEYkChJHzZZUIv0gmq+HOG
CHLyBNsf3BzmBXmwLRIOf54MxwaafysjpHifuYB4Po7G/utvlYqIxg1mBo/KtnMC
Wl2Enmrunei6gN3SAQJBAN94UfraP7lG50OYWhkcso95xpGrCeoFT64dV9I56FhU
q4tD6SSeIXj50hvSH2MQMdFBjsUHPIt+X7zw17ywgwUCQQDNdETg4FD3YT1xolWD
IFEowLlKfNqSaDIQia0dMIzk1/IqOoYhRKH03A97Wae1w/iyMgefyOVvkk+FZUs7
r0rBAkAnZox1wTNJFIJD/cGs+c1V1K+5EUIPO95/oXbRfxpDMLKKPHAH38WhEdME
yrhz++/8qCVnAc6f/akdpA01nJ2NAkEArC1GE9aow8fgADz0wMDygt6P6ZacbZmY
azeVtiKb0KQQM8d75KFpwJQy/UJzQ+aJonw+2282p7vLnJT46XnLgQJALxzSBu0D
hXIDTAMBpon+zgXjiy0ndiigLDPh1jJyCXrH0tBwj3FN+Br3yxbZT2LmE6PLIaJy
rkTCKqlnkG2h+w==
-----END PRIVATE KEY-----
Now, anyone should be able to run the commands I gave above and get the same results. The shell script produces this as its output:
l4BxJyo/GQ0vUF5YR/vO7NtX5Sn/9bGfNHiVGS+W1CMfrwVlCtT0afHERXowx5T8mOiZ90VCJioHMj9Z6ssmfF1SpUbpoo1HYwNBCAfEcIjPLBj4N4KDLpy4gbMZtHEo2B8DZitYLwYDyvkCEudrMiU9b39DqOL+p3pwjJxT5iE=
With no trailing newline of course. The ruby script produces:
l4BxJyo/GQ0vUF5YR/vO7NtX5Sn/9bGfNHiVGS+W1CMfrwVlCtT0afHERXowx5T8mOiZ90VCJioHMj9Z6ssmfF1SpUbpoo1HYwNBCAfEcIjPLBj4N4KDLpy4gbMZtHEo2B8DZitYLwYDyvkCEudrMiU9b39DqOL+p3pwjJxT5iE=
Also without a trailing newline. I've only base 64 encoded the results to make them presentable to humans. That part is not the problem. The real issue is that my invocations of openssl in the ruby script and the shell script are doing different things. In the comments people keep asking about intermediate results. Just to be clear, the only thing that can be considered an "intermediate result" in the raw binary result before the base64 encoding (and it would be both unhelpful and impossible for me to post that here). The openssl command I'm running is just one opaque command (in both scripts), so I don't have access to the SHA256 hash in either case. I suppose that it would be generated with this command:
sha256sum res.tmp
Which gives this at the command line:
688a84cb84ce3b203460a2775248a31d4c3e9590a41ba27134dc342c328b8f9c res.tmp
But I cannot be certain that the hash above is actually any intermediate result of either of the two scripts. Let me know if I can provide more information. Thanks.
Somehow, I ended up mistaken, and it appears that these two scripts are indeed producing the same output. Sorry.
Related
Use Case:
I am currently trying to create a bash script that will be able to upload a file to an AWS S3 bucket. This will be used across a few hundred systems that can not have the AWS CLI, or s3cmd or similar, installed, unfortunately. So I have to resort to bash. I've already generated a zsh script that works as expected for my Mac systems, but now to create on linux.
Work so far:
My code works up until the point that I need to hash the canonical request:
canonical_request_hash=$(echo -n $canonical_request | openssl dgst -sha256)
When I run my code AWS returns the standard
The request signature we calculated does not match the signature you provided. Check your key and signing method.
error.
After looking through the error response, my CanonicalRequest is the same that AWS would generate, but the StringToSign has a different canonical_request_hash.
Here is my code snippet that I generate a different hash than AWS:
# Initial Canoncial Request String
canonical_request="PUT
/$file_name
host:$aws_s3_host.s3.amazonaws.com
x-amz-content-sha256:$file_sha256
x-amz-date:$now_time
host;x-amz-content-sha256;x-amz-date
$file_sha256"
# THE CANONICAL REQUEST ABOVE IS WHAT AWS PRODUCES.
###
# Generate Canonical Request Hash
canonical_request_hash=$(echo -n $canonical_request | openssl dgst -sha256)
###
# THE CANONICAL REQUEST HASH ABOVE IS DIFFERENT THAN WHAT AWS PRODUCES.
# String that will be used to generate AWS Signature
string_to_sign="AWS4-HMAC-SHA256
$now_time
$now_date/$region/s3/aws4_request
$canonical_request_hash"
I have attempted different hash commands:
sha256sum
openssl dgst -sha256
and attempted echo w/ and w/o the -n option.
I tried setting the canonical_request to a file and then hashing the file. Why? No clue, just trying everything I can think of. lol
Any help would be greatly appreciated.
When you leave the $canonical_request variable unquoted, the whitespace changes. Use
printf '%s' "$canonical_request" | openssl ...
Further reading:
3.5.7 Word Splitting in the bash manual
Security implications of forgetting to quote a variable in bash/POSIX shells
Why is printf better than echo?
I have a big audio file (2 hours) foo.mp3. I want to enable a third party P to verify the date of its creation. However P shall only be able to view (hear) small certain parts of foo.mp3. On the date of the creation it is unclear which parts will be provided to P.
So my plan is to split foo.mp3 into small chunks, hash these chunks and publish the hashes somewhere on the net for later verification.
How would a bash script look like that splits foo.mp3 and creates hashes?
You don't have to split the file into chunks. Just publish the SHA256 of the whole file now and when the other party gets access to the file they will be able to see that it has the same SHA256 sum that you published earlier. (I'd recommend using at least SHA256 as the hash algorithm.)
Now if you still decide you have to do it like this for some reason, I suggest using split(1) and then running sha256sum:
mkdir split
split -b8192 foo split/
sha256sum split/*
This creates splits of the file into 8KB chunks in the split/ directory and prints the SHA256 hashes.
1. Splitting the file into several chunks
mp3splt -a -t 0.01 -o foo-#n -d outputDir foo.mp3
2. Create hashes
sha256sum outputDir/* > hashes.sha
3. Document results on the web
The outcome of a published file with sha256 hashes and/or sha256 hashes of big sha256 hashfiles could look like this:)
5b2718841d2f610ce264191760383dca309fd5a6f8c745e7466aa6c157e0b279 Files1.sha
0b568a4e802e9bc29740e8904c09f361c844dc4bd11d8855a188e81672f453ba Files2.sha
d8fec62a53228cf372ce81f4e3fd220b2df7bb5d5f8b144c8ffa4e41a4aeaad4 Camera.zip
ac0e59caa3f91f49824540fa452e285b9e47843f24dda309955a5287db4c89b5 Camerasmall.zip
e1f9b37f15eed99a8814f066021e6d370fd4e346b2779fc6494ee685b9e02467 SoundsSmall.zip
d054a2ac8adf069c2051659c68e38a7b6e26172afb2a6c377f7d067c666f9f6e Sounds.zip
I'd like to migrate this unix script o ruby:
echo -n "loremipsum" | openssl dgst -md5 -hex -sign keys/#02299991.privKey.pem
The result is the following:
(stdin)= 08d9d6496a5146bc1955ad35c884e3b843d441eebc9ed7908b220e9414132dad57dc0f4744e5ec4a9a819e20f4099e2c90186b4684b3b52d1409dd4ca5bc86e7c16dbb64e6cf41d695a7e979744616fe92e5347a7acbb1e1da902d3cfc629927adf3e119f33d2cbc89f90f9494d44becbf93855d09460a67e2615e7d8df7e4f8
I'm using the following code:
key = OpenSSL::PKey::RSA.new(File.read("keys/#02299991.privKey.pem"))
md5_hash = Digest::MD5.hexdigest 'loremipsum' # result: 65a73f29730d3519bd7dd98ab954ed56
key.private_encrypt md5_hash
And this is the problem private_encrypt creates a very different hash from the one that i generated with command line:
"B62\xDA\x80\xF9\xFF]\xCE;\a\xB3)fC\xA8v\x9EM5\xF8Z\xA9\x00\b\xA4\x95 \x84\x8A\xD6:\xDA\xCE\x1D\x01\x9F&\xEB\xD5\xD6\xDF\xC0\v\xD8i\xA0\x86\x8E\e`\x98\xB9\x19\xEC\xA7\x8A\\.\xD1\xCC\xFC\x93\x1C\xFF\xFFh\xAFw\t\xFF\xC8Z\xEC\xBDP\x9C_\x03%\x85:\x04\x1C=\xF3\xAC\xE1\x917TO\x94\xB2\x9Fd`3\x98\x04\x93\xBFS\\\xD1Z\xF9\xBD\x91\xE3\xA1:\xA3B22U_FI'`0i\x8D(\x9B`"
what's wrong?
Your openssl command and your Ruby code do very different things. Your openssl command says "generate an MD5 digest signed with my RSA key and print it in hex format." Your Ruby code says "generate an (unsigned) MD5 digest in hex format and then encrypt it with my RSA key." Digest generation and encryption are not the same thing and are not going to give you the same result.
The Ruby equivalent to your openssl command is this:
data = 'loremipsum'
digest = OpenSSL::Digest::MD5.new
pkey = OpenSSL::PKey::RSA.new(File.read("keys/#02299991.privKey.pem"))
signature = pkey.sign(digest, data)
hex_signature = signature.unpack('H*').first
The first four lines are practically verbatim from the docs for OpenSSL::PKey::PKey#sign. That method returns a binary string (I couldn't find a way to get a hex string directly; if anyone knows a way please leave a comment), so we have to use String#unpack to turn it into a hex string. (unpack returns an array, ergo .first.)
When executing
lines = `gpg --list-keys --with-colons horst`
What Encoding will the string lines have? How do I change how Ruby interprets it?
Background:
I have some Umlauts in some gpg keys, and I get this error when trying to split by newline:
invalid byte sequence in UTF-8
My current workaround is this:
lines.force_encoding('ISO-8859-1')
However, I don't get why this should be ISO-8859-1, as my locale is en_US.UTF-8..
I'm not sure if you still need an answer on this or not but it looks like you'll have to use the --display-charset or –charset option in your gpg command in order to set the name of the native character set. This is used to convert some strings to proper UTF-8 encoding. You shouldn't have to enforce encoding downstream after you've done that.
Check the gpg man page on your server to see which option is available to you.
For example under FreeBSD passwords are stored in /etc/master.passwd like this:
$1$7wtGfwgp$772bEQInetnJKUNtLM0Xt/
The password I used was "Test11". As I understand, $1$ means that it's a hashed MD5 crypt? How can one come up with the final hash "772bEQInetnJKUNtLM0Xt/" if he is aware of salt and password and uses md5sum? As I understand, it should be something like:
$ echo -n $(echo -n 7wtGfwgp)$(echo -n Test11) | md5sum
..but this doesn't quite match up. What am I doing wrong?
PS I'm asking this in order to understand the password hashing system under UNIX-like operating systems.
I'm on Linux and I am not sure whether FreeBSD actually uses the same algorithm, but you can take a look at these sources of information (hope the same is on FreeBSD):
man 3 crypt
https://unix.stackexchange.com/questions/21897/grub-md5-crypt-algorithm
http://s23.org/wiki/Crypt
Based on the last page, this PHP script will produce the output you expect (given your password and salt):
<?php
$password = 'Test11';
$salt = '$1$7wtGfwgp$';
echo 'Crypt hash: ' . crypt($password, $salt) . "\n"
?>
You can do the same using e.g. Python:
import crypt
password = 'Test11'
salt = '$1$7wtGfwgp$'
print(crypt.crypt(password, salt))
based on this Python doc page:
http://docs.python.org/library/crypt.html
Based on the Wikipedia article:
http://en.wikipedia.org/wiki/Crypt_(Unix)
you can see the source of crypt function e.g. here:
http://google.com/codesearch/p#ZWtxA-fyzBo/UnixArchive/PDP-11/Distributions/research/Henry_Spencer_v7/v7.tar.gz%7C118goTAkg2o/usr/src/libc/gen/crypt.c
As a side note, here's a nice online hash generator:
http://insidepro.com/hashes.php?lang=eng
Hope this helps.
Hashing and crypting is something different, even if hashing is a part of crypting ;)
So if you want to crypt it, do it like icyrock posted. md5sum (gmd5sum in freebsd coreutils) does only create a hash sum (RFC 1321) about some input. It's not using this hash to crypt in a further step as (several) crypt codes does.
That's why your code gives you something completely different as result.