I'd like to migrate this unix script o ruby:
echo -n "loremipsum" | openssl dgst -md5 -hex -sign keys/#02299991.privKey.pem
The result is the following:
(stdin)= 08d9d6496a5146bc1955ad35c884e3b843d441eebc9ed7908b220e9414132dad57dc0f4744e5ec4a9a819e20f4099e2c90186b4684b3b52d1409dd4ca5bc86e7c16dbb64e6cf41d695a7e979744616fe92e5347a7acbb1e1da902d3cfc629927adf3e119f33d2cbc89f90f9494d44becbf93855d09460a67e2615e7d8df7e4f8
I'm using the following code:
key = OpenSSL::PKey::RSA.new(File.read("keys/#02299991.privKey.pem"))
md5_hash = Digest::MD5.hexdigest 'loremipsum' # result: 65a73f29730d3519bd7dd98ab954ed56
key.private_encrypt md5_hash
And this is the problem private_encrypt creates a very different hash from the one that i generated with command line:
"B62\xDA\x80\xF9\xFF]\xCE;\a\xB3)fC\xA8v\x9EM5\xF8Z\xA9\x00\b\xA4\x95 \x84\x8A\xD6:\xDA\xCE\x1D\x01\x9F&\xEB\xD5\xD6\xDF\xC0\v\xD8i\xA0\x86\x8E\e`\x98\xB9\x19\xEC\xA7\x8A\\.\xD1\xCC\xFC\x93\x1C\xFF\xFFh\xAFw\t\xFF\xC8Z\xEC\xBDP\x9C_\x03%\x85:\x04\x1C=\xF3\xAC\xE1\x917TO\x94\xB2\x9Fd`3\x98\x04\x93\xBFS\\\xD1Z\xF9\xBD\x91\xE3\xA1:\xA3B22U_FI'`0i\x8D(\x9B`"
what's wrong?
Your openssl command and your Ruby code do very different things. Your openssl command says "generate an MD5 digest signed with my RSA key and print it in hex format." Your Ruby code says "generate an (unsigned) MD5 digest in hex format and then encrypt it with my RSA key." Digest generation and encryption are not the same thing and are not going to give you the same result.
The Ruby equivalent to your openssl command is this:
data = 'loremipsum'
digest = OpenSSL::Digest::MD5.new
pkey = OpenSSL::PKey::RSA.new(File.read("keys/#02299991.privKey.pem"))
signature = pkey.sign(digest, data)
hex_signature = signature.unpack('H*').first
The first four lines are practically verbatim from the docs for OpenSSL::PKey::PKey#sign. That method returns a binary string (I couldn't find a way to get a hex string directly; if anyone knows a way please leave a comment), so we have to use String#unpack to turn it into a hex string. (unpack returns an array, ergo .first.)
Related
Use Case:
I am currently trying to create a bash script that will be able to upload a file to an AWS S3 bucket. This will be used across a few hundred systems that can not have the AWS CLI, or s3cmd or similar, installed, unfortunately. So I have to resort to bash. I've already generated a zsh script that works as expected for my Mac systems, but now to create on linux.
Work so far:
My code works up until the point that I need to hash the canonical request:
canonical_request_hash=$(echo -n $canonical_request | openssl dgst -sha256)
When I run my code AWS returns the standard
The request signature we calculated does not match the signature you provided. Check your key and signing method.
error.
After looking through the error response, my CanonicalRequest is the same that AWS would generate, but the StringToSign has a different canonical_request_hash.
Here is my code snippet that I generate a different hash than AWS:
# Initial Canoncial Request String
canonical_request="PUT
/$file_name
host:$aws_s3_host.s3.amazonaws.com
x-amz-content-sha256:$file_sha256
x-amz-date:$now_time
host;x-amz-content-sha256;x-amz-date
$file_sha256"
# THE CANONICAL REQUEST ABOVE IS WHAT AWS PRODUCES.
###
# Generate Canonical Request Hash
canonical_request_hash=$(echo -n $canonical_request | openssl dgst -sha256)
###
# THE CANONICAL REQUEST HASH ABOVE IS DIFFERENT THAN WHAT AWS PRODUCES.
# String that will be used to generate AWS Signature
string_to_sign="AWS4-HMAC-SHA256
$now_time
$now_date/$region/s3/aws4_request
$canonical_request_hash"
I have attempted different hash commands:
sha256sum
openssl dgst -sha256
and attempted echo w/ and w/o the -n option.
I tried setting the canonical_request to a file and then hashing the file. Why? No clue, just trying everything I can think of. lol
Any help would be greatly appreciated.
When you leave the $canonical_request variable unquoted, the whitespace changes. Use
printf '%s' "$canonical_request" | openssl ...
Further reading:
3.5.7 Word Splitting in the bash manual
Security implications of forgetting to quote a variable in bash/POSIX shells
Why is printf better than echo?
I'm trying to figure out the equivalent shell script of a small ruby script I have. This is the ruby script:
require 'openssl'
require 'base64'
k = OpenSSL::PKey::RSA.new(File.read("key.pem"))
res = File.read("res.tmp")
digest = OpenSSL::Digest::SHA256.new
signature = k.sign(digest,res)
File.write("foo1.txt",Base64.strict_encode64(signature))
That's it. It takes some data, gets the SHA256 hash of it, and then signs that with a private key I have. I think that the equivalent command on the terminal should be:
openssl sha -sha256 -sign key.pem < res.tmp | base64 -w 0 > foo2.txt
But these do not produce the same output. Could anyone enlighten me as to why?
-- EDIT --
I'm adding some more information so that people can try to reproduce this on their computers if so desired. The contents of res.tmp are:
This is some sample text.
This is some sample text.
This is some sample text.
This is some more sample text.
This is some more sample text.
This is some more sample text.
The private key (which is not a private key that is actually used for any production system, just clarifying that) is:
-----BEGIN PRIVATE KEY-----
MIICdgIBADANBgkqhkiG9w0BAQEFAASCAmAwggJcAgEAAoGBALNY2EtJTj78tvnL
G1v+5HFqn8mZ85AToUqj/n86/eymcMqEjskHAoVmU9FOr+ZsnSxopNvUVLCzDkv6
w88tUtHGblzFtzJpQrxvtH8VCd2BNUc6Esxx7lmQeCxJMMPnmY2ZkDZuH6L+UsVL
DbM7LCvJyluo/A3AP68G9YL+XTjFAgMBAAECgYBkJzsyX894UTwFJq0ypJcB1x9A
P97KGIw72HTorBLdMt1N2tS54lZAFLK98gk8zm6/O/jEYkChJHzZZUIv0gmq+HOG
CHLyBNsf3BzmBXmwLRIOf54MxwaafysjpHifuYB4Po7G/utvlYqIxg1mBo/KtnMC
Wl2Enmrunei6gN3SAQJBAN94UfraP7lG50OYWhkcso95xpGrCeoFT64dV9I56FhU
q4tD6SSeIXj50hvSH2MQMdFBjsUHPIt+X7zw17ywgwUCQQDNdETg4FD3YT1xolWD
IFEowLlKfNqSaDIQia0dMIzk1/IqOoYhRKH03A97Wae1w/iyMgefyOVvkk+FZUs7
r0rBAkAnZox1wTNJFIJD/cGs+c1V1K+5EUIPO95/oXbRfxpDMLKKPHAH38WhEdME
yrhz++/8qCVnAc6f/akdpA01nJ2NAkEArC1GE9aow8fgADz0wMDygt6P6ZacbZmY
azeVtiKb0KQQM8d75KFpwJQy/UJzQ+aJonw+2282p7vLnJT46XnLgQJALxzSBu0D
hXIDTAMBpon+zgXjiy0ndiigLDPh1jJyCXrH0tBwj3FN+Br3yxbZT2LmE6PLIaJy
rkTCKqlnkG2h+w==
-----END PRIVATE KEY-----
Now, anyone should be able to run the commands I gave above and get the same results. The shell script produces this as its output:
l4BxJyo/GQ0vUF5YR/vO7NtX5Sn/9bGfNHiVGS+W1CMfrwVlCtT0afHERXowx5T8mOiZ90VCJioHMj9Z6ssmfF1SpUbpoo1HYwNBCAfEcIjPLBj4N4KDLpy4gbMZtHEo2B8DZitYLwYDyvkCEudrMiU9b39DqOL+p3pwjJxT5iE=
With no trailing newline of course. The ruby script produces:
l4BxJyo/GQ0vUF5YR/vO7NtX5Sn/9bGfNHiVGS+W1CMfrwVlCtT0afHERXowx5T8mOiZ90VCJioHMj9Z6ssmfF1SpUbpoo1HYwNBCAfEcIjPLBj4N4KDLpy4gbMZtHEo2B8DZitYLwYDyvkCEudrMiU9b39DqOL+p3pwjJxT5iE=
Also without a trailing newline. I've only base 64 encoded the results to make them presentable to humans. That part is not the problem. The real issue is that my invocations of openssl in the ruby script and the shell script are doing different things. In the comments people keep asking about intermediate results. Just to be clear, the only thing that can be considered an "intermediate result" in the raw binary result before the base64 encoding (and it would be both unhelpful and impossible for me to post that here). The openssl command I'm running is just one opaque command (in both scripts), so I don't have access to the SHA256 hash in either case. I suppose that it would be generated with this command:
sha256sum res.tmp
Which gives this at the command line:
688a84cb84ce3b203460a2775248a31d4c3e9590a41ba27134dc342c328b8f9c res.tmp
But I cannot be certain that the hash above is actually any intermediate result of either of the two scripts. Let me know if I can provide more information. Thanks.
Somehow, I ended up mistaken, and it appears that these two scripts are indeed producing the same output. Sorry.
I'm trying to create an initial user in my postgresql database on deployment. Apparently I need to generate a pbkdf2:sha1 key of the password that I want for the user.
For example, say I wanted to add an admin account with username "admin" and password "admin" I would need to hash the password "admin" before inserting it into the table I believe.
Can anyone tell me how I might do this in bash, for example, I tried something along these lines but it didn't seem to work:
ADMIN_PASSWORD=admin
openssl dgst -sha1 -hmac "$ADMIN_PASSWORD"
EDIT: This is a password for postgres database, that's why it needs to be hashed before it can be inserted. I want to achieve this hashing in bash and not python.
Input should be passed through stdin. Do this:
ADMIN_PASSWORD=admin
echo -n "${ADMIN_PASSWORD}" | openssl dgst -sha1 -hmac
And you will get output
d033e22ae348aeb5660fc2140aec35850c4da997
on stdout. If you'd rather save it to a variable to be referenced later, do
ADMIN_PASSWORD=admin
SHA1SUM=$(echo "${ADMIN_PASSWORD}" | openssl dgst -sha1 -hmac)
And later refer to the hash as ${SHA1SUM}.
Note: curly braces in ${ADMIN_PASSWORD} and ${SHA1SUM} are optional.
Since this is a one-off task, the hash value will never change - it's a constant, so code it that way:
discover what hash your applicatiin would create for the "admin" user with a password of "admin", and hard-code it into your bash script.
Why won't you have PostgreSQL generate the hash for you? The functions crypt() and gen_salt() come with some examples, see the documentation.
When executing
lines = `gpg --list-keys --with-colons horst`
What Encoding will the string lines have? How do I change how Ruby interprets it?
Background:
I have some Umlauts in some gpg keys, and I get this error when trying to split by newline:
invalid byte sequence in UTF-8
My current workaround is this:
lines.force_encoding('ISO-8859-1')
However, I don't get why this should be ISO-8859-1, as my locale is en_US.UTF-8..
I'm not sure if you still need an answer on this or not but it looks like you'll have to use the --display-charset or –charset option in your gpg command in order to set the name of the native character set. This is used to convert some strings to proper UTF-8 encoding. You shouldn't have to enforce encoding downstream after you've done that.
Check the gpg man page on your server to see which option is available to you.
For example under FreeBSD passwords are stored in /etc/master.passwd like this:
$1$7wtGfwgp$772bEQInetnJKUNtLM0Xt/
The password I used was "Test11". As I understand, $1$ means that it's a hashed MD5 crypt? How can one come up with the final hash "772bEQInetnJKUNtLM0Xt/" if he is aware of salt and password and uses md5sum? As I understand, it should be something like:
$ echo -n $(echo -n 7wtGfwgp)$(echo -n Test11) | md5sum
..but this doesn't quite match up. What am I doing wrong?
PS I'm asking this in order to understand the password hashing system under UNIX-like operating systems.
I'm on Linux and I am not sure whether FreeBSD actually uses the same algorithm, but you can take a look at these sources of information (hope the same is on FreeBSD):
man 3 crypt
https://unix.stackexchange.com/questions/21897/grub-md5-crypt-algorithm
http://s23.org/wiki/Crypt
Based on the last page, this PHP script will produce the output you expect (given your password and salt):
<?php
$password = 'Test11';
$salt = '$1$7wtGfwgp$';
echo 'Crypt hash: ' . crypt($password, $salt) . "\n"
?>
You can do the same using e.g. Python:
import crypt
password = 'Test11'
salt = '$1$7wtGfwgp$'
print(crypt.crypt(password, salt))
based on this Python doc page:
http://docs.python.org/library/crypt.html
Based on the Wikipedia article:
http://en.wikipedia.org/wiki/Crypt_(Unix)
you can see the source of crypt function e.g. here:
http://google.com/codesearch/p#ZWtxA-fyzBo/UnixArchive/PDP-11/Distributions/research/Henry_Spencer_v7/v7.tar.gz%7C118goTAkg2o/usr/src/libc/gen/crypt.c
As a side note, here's a nice online hash generator:
http://insidepro.com/hashes.php?lang=eng
Hope this helps.
Hashing and crypting is something different, even if hashing is a part of crypting ;)
So if you want to crypt it, do it like icyrock posted. md5sum (gmd5sum in freebsd coreutils) does only create a hash sum (RFC 1321) about some input. It's not using this hash to crypt in a further step as (several) crypt codes does.
That's why your code gives you something completely different as result.