For example under FreeBSD passwords are stored in /etc/master.passwd like this:
$1$7wtGfwgp$772bEQInetnJKUNtLM0Xt/
The password I used was "Test11". As I understand, $1$ means that it's a hashed MD5 crypt? How can one come up with the final hash "772bEQInetnJKUNtLM0Xt/" if he is aware of salt and password and uses md5sum? As I understand, it should be something like:
$ echo -n $(echo -n 7wtGfwgp)$(echo -n Test11) | md5sum
..but this doesn't quite match up. What am I doing wrong?
PS I'm asking this in order to understand the password hashing system under UNIX-like operating systems.
I'm on Linux and I am not sure whether FreeBSD actually uses the same algorithm, but you can take a look at these sources of information (hope the same is on FreeBSD):
man 3 crypt
https://unix.stackexchange.com/questions/21897/grub-md5-crypt-algorithm
http://s23.org/wiki/Crypt
Based on the last page, this PHP script will produce the output you expect (given your password and salt):
<?php
$password = 'Test11';
$salt = '$1$7wtGfwgp$';
echo 'Crypt hash: ' . crypt($password, $salt) . "\n"
?>
You can do the same using e.g. Python:
import crypt
password = 'Test11'
salt = '$1$7wtGfwgp$'
print(crypt.crypt(password, salt))
based on this Python doc page:
http://docs.python.org/library/crypt.html
Based on the Wikipedia article:
http://en.wikipedia.org/wiki/Crypt_(Unix)
you can see the source of crypt function e.g. here:
http://google.com/codesearch/p#ZWtxA-fyzBo/UnixArchive/PDP-11/Distributions/research/Henry_Spencer_v7/v7.tar.gz%7C118goTAkg2o/usr/src/libc/gen/crypt.c
As a side note, here's a nice online hash generator:
http://insidepro.com/hashes.php?lang=eng
Hope this helps.
Hashing and crypting is something different, even if hashing is a part of crypting ;)
So if you want to crypt it, do it like icyrock posted. md5sum (gmd5sum in freebsd coreutils) does only create a hash sum (RFC 1321) about some input. It's not using this hash to crypt in a further step as (several) crypt codes does.
That's why your code gives you something completely different as result.
Related
Use Case:
I am currently trying to create a bash script that will be able to upload a file to an AWS S3 bucket. This will be used across a few hundred systems that can not have the AWS CLI, or s3cmd or similar, installed, unfortunately. So I have to resort to bash. I've already generated a zsh script that works as expected for my Mac systems, but now to create on linux.
Work so far:
My code works up until the point that I need to hash the canonical request:
canonical_request_hash=$(echo -n $canonical_request | openssl dgst -sha256)
When I run my code AWS returns the standard
The request signature we calculated does not match the signature you provided. Check your key and signing method.
error.
After looking through the error response, my CanonicalRequest is the same that AWS would generate, but the StringToSign has a different canonical_request_hash.
Here is my code snippet that I generate a different hash than AWS:
# Initial Canoncial Request String
canonical_request="PUT
/$file_name
host:$aws_s3_host.s3.amazonaws.com
x-amz-content-sha256:$file_sha256
x-amz-date:$now_time
host;x-amz-content-sha256;x-amz-date
$file_sha256"
# THE CANONICAL REQUEST ABOVE IS WHAT AWS PRODUCES.
###
# Generate Canonical Request Hash
canonical_request_hash=$(echo -n $canonical_request | openssl dgst -sha256)
###
# THE CANONICAL REQUEST HASH ABOVE IS DIFFERENT THAN WHAT AWS PRODUCES.
# String that will be used to generate AWS Signature
string_to_sign="AWS4-HMAC-SHA256
$now_time
$now_date/$region/s3/aws4_request
$canonical_request_hash"
I have attempted different hash commands:
sha256sum
openssl dgst -sha256
and attempted echo w/ and w/o the -n option.
I tried setting the canonical_request to a file and then hashing the file. Why? No clue, just trying everything I can think of. lol
Any help would be greatly appreciated.
When you leave the $canonical_request variable unquoted, the whitespace changes. Use
printf '%s' "$canonical_request" | openssl ...
Further reading:
3.5.7 Word Splitting in the bash manual
Security implications of forgetting to quote a variable in bash/POSIX shells
Why is printf better than echo?
I'm trying to figure out the equivalent shell script of a small ruby script I have. This is the ruby script:
require 'openssl'
require 'base64'
k = OpenSSL::PKey::RSA.new(File.read("key.pem"))
res = File.read("res.tmp")
digest = OpenSSL::Digest::SHA256.new
signature = k.sign(digest,res)
File.write("foo1.txt",Base64.strict_encode64(signature))
That's it. It takes some data, gets the SHA256 hash of it, and then signs that with a private key I have. I think that the equivalent command on the terminal should be:
openssl sha -sha256 -sign key.pem < res.tmp | base64 -w 0 > foo2.txt
But these do not produce the same output. Could anyone enlighten me as to why?
-- EDIT --
I'm adding some more information so that people can try to reproduce this on their computers if so desired. The contents of res.tmp are:
This is some sample text.
This is some sample text.
This is some sample text.
This is some more sample text.
This is some more sample text.
This is some more sample text.
The private key (which is not a private key that is actually used for any production system, just clarifying that) is:
-----BEGIN PRIVATE KEY-----
MIICdgIBADANBgkqhkiG9w0BAQEFAASCAmAwggJcAgEAAoGBALNY2EtJTj78tvnL
G1v+5HFqn8mZ85AToUqj/n86/eymcMqEjskHAoVmU9FOr+ZsnSxopNvUVLCzDkv6
w88tUtHGblzFtzJpQrxvtH8VCd2BNUc6Esxx7lmQeCxJMMPnmY2ZkDZuH6L+UsVL
DbM7LCvJyluo/A3AP68G9YL+XTjFAgMBAAECgYBkJzsyX894UTwFJq0ypJcB1x9A
P97KGIw72HTorBLdMt1N2tS54lZAFLK98gk8zm6/O/jEYkChJHzZZUIv0gmq+HOG
CHLyBNsf3BzmBXmwLRIOf54MxwaafysjpHifuYB4Po7G/utvlYqIxg1mBo/KtnMC
Wl2Enmrunei6gN3SAQJBAN94UfraP7lG50OYWhkcso95xpGrCeoFT64dV9I56FhU
q4tD6SSeIXj50hvSH2MQMdFBjsUHPIt+X7zw17ywgwUCQQDNdETg4FD3YT1xolWD
IFEowLlKfNqSaDIQia0dMIzk1/IqOoYhRKH03A97Wae1w/iyMgefyOVvkk+FZUs7
r0rBAkAnZox1wTNJFIJD/cGs+c1V1K+5EUIPO95/oXbRfxpDMLKKPHAH38WhEdME
yrhz++/8qCVnAc6f/akdpA01nJ2NAkEArC1GE9aow8fgADz0wMDygt6P6ZacbZmY
azeVtiKb0KQQM8d75KFpwJQy/UJzQ+aJonw+2282p7vLnJT46XnLgQJALxzSBu0D
hXIDTAMBpon+zgXjiy0ndiigLDPh1jJyCXrH0tBwj3FN+Br3yxbZT2LmE6PLIaJy
rkTCKqlnkG2h+w==
-----END PRIVATE KEY-----
Now, anyone should be able to run the commands I gave above and get the same results. The shell script produces this as its output:
l4BxJyo/GQ0vUF5YR/vO7NtX5Sn/9bGfNHiVGS+W1CMfrwVlCtT0afHERXowx5T8mOiZ90VCJioHMj9Z6ssmfF1SpUbpoo1HYwNBCAfEcIjPLBj4N4KDLpy4gbMZtHEo2B8DZitYLwYDyvkCEudrMiU9b39DqOL+p3pwjJxT5iE=
With no trailing newline of course. The ruby script produces:
l4BxJyo/GQ0vUF5YR/vO7NtX5Sn/9bGfNHiVGS+W1CMfrwVlCtT0afHERXowx5T8mOiZ90VCJioHMj9Z6ssmfF1SpUbpoo1HYwNBCAfEcIjPLBj4N4KDLpy4gbMZtHEo2B8DZitYLwYDyvkCEudrMiU9b39DqOL+p3pwjJxT5iE=
Also without a trailing newline. I've only base 64 encoded the results to make them presentable to humans. That part is not the problem. The real issue is that my invocations of openssl in the ruby script and the shell script are doing different things. In the comments people keep asking about intermediate results. Just to be clear, the only thing that can be considered an "intermediate result" in the raw binary result before the base64 encoding (and it would be both unhelpful and impossible for me to post that here). The openssl command I'm running is just one opaque command (in both scripts), so I don't have access to the SHA256 hash in either case. I suppose that it would be generated with this command:
sha256sum res.tmp
Which gives this at the command line:
688a84cb84ce3b203460a2775248a31d4c3e9590a41ba27134dc342c328b8f9c res.tmp
But I cannot be certain that the hash above is actually any intermediate result of either of the two scripts. Let me know if I can provide more information. Thanks.
Somehow, I ended up mistaken, and it appears that these two scripts are indeed producing the same output. Sorry.
I'm trying to create an initial user in my postgresql database on deployment. Apparently I need to generate a pbkdf2:sha1 key of the password that I want for the user.
For example, say I wanted to add an admin account with username "admin" and password "admin" I would need to hash the password "admin" before inserting it into the table I believe.
Can anyone tell me how I might do this in bash, for example, I tried something along these lines but it didn't seem to work:
ADMIN_PASSWORD=admin
openssl dgst -sha1 -hmac "$ADMIN_PASSWORD"
EDIT: This is a password for postgres database, that's why it needs to be hashed before it can be inserted. I want to achieve this hashing in bash and not python.
Input should be passed through stdin. Do this:
ADMIN_PASSWORD=admin
echo -n "${ADMIN_PASSWORD}" | openssl dgst -sha1 -hmac
And you will get output
d033e22ae348aeb5660fc2140aec35850c4da997
on stdout. If you'd rather save it to a variable to be referenced later, do
ADMIN_PASSWORD=admin
SHA1SUM=$(echo "${ADMIN_PASSWORD}" | openssl dgst -sha1 -hmac)
And later refer to the hash as ${SHA1SUM}.
Note: curly braces in ${ADMIN_PASSWORD} and ${SHA1SUM} are optional.
Since this is a one-off task, the hash value will never change - it's a constant, so code it that way:
discover what hash your applicatiin would create for the "admin" user with a password of "admin", and hard-code it into your bash script.
Why won't you have PostgreSQL generate the hash for you? The functions crypt() and gen_salt() come with some examples, see the documentation.
We have a mail server and I am trying to write a script that will find all users with similar names to avoid malicious users from impersonating legitimate users. For example, a legit user may have the name of james2014#domain.com but a malicious user may register as james20l4#domain.com. The difference, if you notice carefully, is that I replaced the number 'one' with the letter 'l' (el). So I am trying to write something that can consult my /var/vmail/domain/* and find similar names and alert me (the administrator). I will then take the necessary steps to do what I need. Really appreciate any help.
One hacky way to do this is to derive "normalized" versions of your usernames, put those in an associative array as keys mapping to the original input, and use those to find problems.
The example I posted below uses bash associative arrays to store the mapping from normalized name to original name, and tr to switch some characters for other characters (and delete other characters entirely).
I'm assuming that your list of users will fit into memory; you'll also need to tweak the mapping of modified and removed characters to hit your favorite balance between effectiveness and false positives. If your list can't fit in memory, you can use a single file or the filesystem to approximate it, but honestly if you're processing that many names you're probably better off with a non-shell programming language.
Input:
doc
dopey
james2014
happy
bashful
grumpy
james20l4
sleepy
james.2014
sneezy
Script:
#!/bin/bash
# stdin: A list of usernames. stdout: Pairs of names that match.
CHARS_TO_REMOVE="._\\- "
CHARS_TO_MAP_FROM="OISZql"
CHARS_TO_MAP_TO="0152g1"
normalize() {
# stdin: A word. stdout: A modified version of the same word.
exec tr "$CHARS_TO_MAP_FROM" "$CHARS_TO_MAP_TO" \
| tr --delete "$CHARS_TO_REMOVE" \
| tr "A-Z" "a-z"
}
declare -A NORMALIZED_NAMES
while read NAME; do
NORMALIZED_NAME=$(normalize <<< "$NAME")
# -n tests for non-empty strings, as it would be if the name were set already.
if [[ -n ${NORMALIZED_NAMES[$NORMALIZED_NAME]} ]]; then
# This name has been seen before! Print both of them.
echo "${NORMALIZED_NAMES[$NORMALIZED_NAME]} $NAME"
else
# This name has not been seen before. Store it.
NORMALIZED_NAMES["$NORMALIZED_NAME"]="$NAME"
fi
done
Output:
james2014 james20l4
james2014 james.2014
I was trying to do do some more shell scripting in the last few days.
where I am trying to calculate the value of a function.
To start with I get an
echo "blah"
read blub
md5=`md5sum $blub | cut -d ' ' -f 1`
echo $md5
echo "secretkey"
read $SK
For example, If get the values like that.
Is it possible at-all to do a calculation withGPG like this
GPG(VALUE1,VALUE2) in my case it is GPG(md5,sk)
Later on I would like to take a modular function over that value. Which is possible via the shell script.
If its possible , May I ask how is it possible to do in GPG. or is there any other better public generating algorithm that can do this?
Thanks!
If I understand you correctly, you are trying to obtain a digest (hash, checksum) of the MD5 + secret key pair (much like the MD5 is a digest of the blub).
In any case, GPG won't help you here. It's mostly a tool for encryption and signing with PGP. It can do symmetric encryption of files as well but AFAICS it doesn't do stand-alone digests, and certainly doesn't do them in decimal form.
The simplest solution seems to be using one of the common shell checksumming tools which actually have numeric output — for example cksum.
Those tools take only a single blob as an argument, so you probably want to simply concatenate the input
I think a working example would be:
gpg=$(echo "$md5$sk" | cksum | cut -d' ' -f1)
Then gpg will contain some decimal value, unique for each md5 + secret key pair.