I'm trying to do a very simple thing, namely using a 64-bit password and a 64-bit plaintext (both in hex) and encrypt it with simple old DES.
my script looks like this:
plaintext=`echo -n "$2" | sed 's/\(..\)/\\\x\1/g'`
key=$1
printf "$plaintext" | openssl enc -nosalt -e -des -nopad -K "$key" -iv "0000000000000000" | od --format=x1 --width=32 --address-radix=n | sed 's/ //g'
I execute and get the following result:
./des_enc 5B5A57676A56676E 675A69675E5A6B5A
b617e2c84a4fba2149dd7132433031392257b99d9284b1031c351c15825aca52
The problem is there's too much data coming back from openssl, I expect to only get 64-bits of data instead I get 512. I don't know how to explicit request a 64-bit version of DES, is it even possible?
Note: The values used above are from "H. Katzan, The Standard Data Encryption Algorithm, pp75-94, Petrocelli Books Inc., New York, 1977" is:
Key: 5B5A57676A56676E
Plaintext: 675A69675E5A6B5A
Ciphertext: 974AFFBF86022D1F
Use -des-ecb. Also, xxd makes this pipeline much cleaner, if you have it handy (it's part of the vim package):
sh % echo 675A69675E5A6B5A | xxd -r -ps | openssl enc -des-ecb -nopad -K 5B5A57676A56676E | xxd -ps
974affbf86022d1f
Related
Why would these respective bash and powershell scripts return different signatures:
## (Ubuntu) Bash
now='Fri, 26 Jul 2019 12:32:36 -0400'
bucket="mybucket"
path="/"
awsKey="MSB0M3K06ELMOI65QXI1"
awsSsecret="706Fdj+LFKf8pf/2Wh5V8Q8jbgGUUQo3xSXr5sbt"
## Create the signature
message="GET\n\n\n${now}\n/${bucket}${path}"
messageSignature=$(echo -en "${message}" | openssl sha1 -hmac "${awsSsecret}" -binary | openssl base64)
echo $messageSignature
Returns >> 5oJM2L06LeTcbYSfN3fDFZ9yt5k=
## Powershell
$OutputEncoding = [Text.UTF8Encoding]::UTF8
$now='Fri, 26 Jul 2019 12:32:36 -0400'
$bucket="mybucket"
$path="/"
$awsKey="MSB0M3K06ELMOI65QXI1"
$awsSsecret="706Fdj+LFKf8pf/2Wh5V8Q8jbgGUUQo3xSXr5sbt"
## Create the signature
$message="GET\n\n\n${now}\n/${bucket}${path}"
$messageSignature=$(echo -en "${message}" | openssl sha1 -hmac "${awsSsecret}" -binary | openssl base64)
echo $messageSignature
Returns >> 77u/W8O5RwJeBsOodDddXeKUlCQD4pSUduKVrD3ilIxADQo=
]2
On Ubuntu, my shell is running "en_US.UTF-8".
I've run into the case where the signature is different on different systems: AIX, Windows w/Ubuntu, Windows w/Powershell, etc. I'm trying to figure out why.
There are at least three problems here: Powershell pipelines aren't binary-safe, you have the wrong variable name, and echo isn't portable/reliable.
At least according to this question, you can't pipe binary data in Powershell. The output of openssl (which is raw binary data) is getting treated as UTF-8 text (presumably due to $OutputEncoding), and mangled in the process. You can tell by decoding from base64 and looking in hex instead:
$ echo '77u/W8O5RwJeBsOodDddXeKUlCQD4pSUduKVrD3ilIxADQo=' | base64 -D | xxd
00000000: efbb bf5b c3b9 4702 5e06 c3a8 7437 5d5d ...[..G.^...t7]]
00000010: e294 9424 03e2 9494 76e2 95ac 3de2 948c ...$....v...=...
00000020: 400d 0a #..
It starts with EF BB BF, which is a UTF-8 byte order mark; and it ends with 0D 0A, which is a DOS/Windows line ending (ASCII carriage return and linefeed characters). Something else bad is happening as well, since it's much too long for a sha1 hash, even if you account for the BOM and line ending.
The output of echo is probably getting mangled similarly, so even if the hash wasn't mangled it'd be the hash of the wrong byte sequence.
See this question (and its answer) for an example of using Powershell's own tools to compute the HMAC-SHA1 of a string.
The message to be signed is in $message, but you actually sign $string_to_sign, which is undefined. The Ubuntu result is the correct HMAC-SHA1 for the null string:
$ </dev/null openssl sha1 -hmac "706Fdj+LFKf8pf/2Wh5V8Q8jbgGUUQo3xSXr5sbt" -binary | openssl base64
NCRRWG4nL9sN8QMrdmCPmUvNlYA=
As Lorinczy Zsigmond pointed out, echo -en isn't predictable. Under some implementations/OSes/shells/compile & runtime options/phases of moon/etc, it might or might not print "-en" as part of its output, and maybe also print a newline (or something) at the end. Use printf instead:
printf 'GET\n\n\n%s\n/%s%s' "${now}" "${bucket}" "${path}" | openssl sha1 ...
Or (in bash, but not all other shells):
printf -v message 'GET\n\n\n%s\n/%s%s' "${now}" "${bucket}" "${path}"
printf '%s' "$message" | openssl sha1 ...
Or (bash again):
nl=$'\n'
message="GET${nl}${nl}${nl}${now}${nl}/${bucket}${path}"
printf '%s' "$message" | openssl sha1 ...
Providing a summary for anyone who stumbled onto this thread. The problem problems were....
Cannot pass binary data through the pipeline as noted by Gordon Davisson
Need to represent newlines with "`n" instead of "\n"
The Powershell script that ultimately replicated my results in Ubuntu/Bash was...
$now='Fri, 26 Jul 2019 12:32:36 -0400'
$bucket="mybucket"
$path="/"
$awsKey="MSB0M3K06ELMOI65QXI1"
$awsSsecret="706Fdj+LFKf8pf/2Wh5V8Q8jbgGUUQo3xSXr5sbt"
$message="GET`n`n`n${now}`n/${bucket}${path}"
## Create the signature
$sha = [System.Security.Cryptography.KeyedHashAlgorithm]::Create("HMACSHA1")
$sha.Key = [System.Text.Encoding]::UTF8.Getbytes($awsSsecret)
[Convert]::Tobase64String($sha.ComputeHash([System.Text.Encoding]::UTF8.Getbytes($message)))
While it's a better idea to use printf over echo on Linux/Unix systems, that difference wasn't material in this use-case.
I'm using a macOS Sierra and want to sha256 each line in a file.
File:
test
example
sample
Intended output:
9F86D081884C7D659A2FEAA0C55AD015A3BF4F1B2B0B822CD15D6C15B0F00A08
50D858E0985ECC7F60418AAF0CC5AB587F42C2570A884095A9E8CCACD0F6545C
AF2BDBE1AA9B6EC1E2ADE1D694F41FC71A831D0268E9891562113D8A62ADD1BF
I've found a few ways to do this on Ubuntu, but Mac doesn't seem to have sha256 installed on it. From what is sounds like, you have to use something similar to shasum 256 but nothing seems to be working.
openssl dgst is available out-of-the-box on MacOS (and pretty much everywhere else as well), and can be easily combined with BashFAQ #1 practices for iterating line-by-line:
### INEFFICIENT SOLUTION
hashlines() {
while IFS= read -r line; do
printf '%s' "$line" \
| openssl dgst -sha256 \
| tr '[[:lower:]]' '[[:upper:]]' \
| sed -Ee 's/.*=[[:space:]]//'
done
}
That said, this is quite wildly inefficient when run on large files. If you need something that performs well, I'd write this in Python instead. Wrapped in the same shell function, with the same calling convention, that might look like:
### FAST SOLUTION
hashlines() {
python -c '
import sys, hashlib
for line in sys.stdin:
print(hashlib.sha256(line.rstrip("\n")).hexdigest().upper())'
}
In either case, usage is just hashlines < filename or hashlines <<< $'line one\nline two'.
I am trying to decrypt a file (part444.txt) with message:
y2EdLtmNQsZkvwwf8jf3fM6c1thfzF0sQfblayGIBik=
This is base64 encoded encrypted text under 128 bit AES in CBC mode. It is not padded. The IV is the first 16 bytes of the encrypted text and the key is h4ckth1sk3yp4d16.
I know that people received the bad magic number error from problems with Base64 but now I get the "error reading input file" and not sure where to go from here.
I have tried:
openssl enc -base64 -d part444.txt | openssl aes-128-cbc -d -k h4ckth1sk3yp4d16
Why am I encountering the errors "bad magic number" and "error reading input file"?
This is sort of a pain to do with openssl, because openssl's encryption makes assumptions about padding and deriving a salted key from the entered password that you have to deliberately turn off.
It's much easier to do in python with say PyCrypto, where these assumptions aren't made.
>>> import base64
>>> data = base64.b64decode('y2EdLtmNQsZkvwwf8jf3fM6c1thfzF0sQfblayGIBik=')
>>> from Crypto.Cipher import AES
>>> aes_crypter = AES.new('h4ckth1sk3yp4d16', AES.MODE_CBC, data[:16])
>>> aes_crypter.decrypt(data[16:]) # this gives the encrypted secret.
It is possible to do this with openssl, but you have to read the base64 encoded data -- take out the first 16 bytes and remember it as your $IV (after encoding it back to hex that openssl expects), start reading all the bytes after the first 16 and remember it as the $CIPHERTEXT (and say re-encode in base64). Similar for the $KEY, you have to convert it from ASCII to bytes in hex. Assuming you stored these in variables, then the following would work:
IV=`base64 -d part444.txt | xxd -p -l 16`
CIPHERTEXT=`base64 -d part444.txt | cut -b 17- | base64`
KEY=`echo -n h4ckth1sk3yp4d16 |xxd -p`
echo $CIPHERTEXT | openssl aes-128-cbc -d -a -nopad -K $KEY -iv $IV && echo ""
Note base64 -d decodes base64 to binary (using base64 from GNU coreutils; on BSD replace with base64 -D), base64 b64 encodes binary data, cut -b 17- reads from the 17th byte of data to the end of the file, and xxd -p converts binary to hex.
What would be the easiest way to convert the text produced by utilities such as sha512sum into a binary file?
I'd like to convert hex string like 77f4de214a5423e3c7df8704f65af57c39c55f08424c75c0931ab09a5bfdf49f5f939f2caeff1e0113d9b3d6363583e4830cf40787100b750cde76f00b8cd3ec (example produced by sha512sum) into a binary file (64 bytes long), in which each byte's value would be equivalent to a pair of letters/digits from the string. I'm looking for a solution that would require minimal amount of tools, so I'd be happy if this can be done easily with bash, sed or some utility from coreutils. I'd rather avoid xxd, as this doesn't seem to handle such string anyway (I'd have to add "addresses" and some whitespace).
I need the hash as a binary file, to convert it into an object file and link with the application that I'm writing. If there's another way to embed such string (in a binary form!) into application (via an array or whatever) - it's also a solution for me.
A bit of sed and echo might do:
for i in $(echo 77f4de214a5423e3c7df8704f65af57c39c55f08424c75c0931ab09a5bfdf49f5f939f2caeff1e0113d9b3d6363583e4830cf40787100b750cde76f00b8cd3ec | sed 's/../& /g'); do
echo -ne "\x$i"
done > output.bin
The sed command is splitting the hex string into bytes and the echo shows it as hexadecimal character.
Or in a shorter form with sha512sum output, as suggested in the comment:
echo -ne "$(sha512sum some-file.txt | sed 's/ .*$//; s/../\\x&/g')"
How about perl:
<<<77f4de214a5423e3c7df8704f65af57c39c55f08424c75c0931ab09a5bfdf49f5f939f2caeff1e0113d9b3d6363583e4830cf40787100b750cde76f00b8cd3ec \
perl -e 'print pack "H*", <STDIN>' > hash.bin
If you have openssl in your system and want a sha512 hash in binary form, you can use this:
openssl dgst -sha512 -binary somefile.txt
If you have node:
node -e "var fs = require('fs'); fs.writeFileSync('binary', new Buffer('77f4de214a5423e3c7df8704f65af57c39c55f08424c75c0931ab09a5bfdf49f5f939f2caeff1e0113d9b3d6363583e4830cf40787100b750cde76f00b8cd3ec', 'hex'))"
s="77f4de214a5423e3c7df8704f65af57c39c55f08424c75c0931ab09a5bfdf49f5f939f2caeff1e0113d9b3d6363583e4830cf40787100b750cde76f00b8cd3ec";
echo -n $s | xxd -r -p > file.bin
1 File(s) 64 bytes
Tested on Ubuntu 16.04.7
I am trying to calculate the digests of a file using md5 algorithm. I am asked to format the output to be in binary not hex. So this is my command in terminal (I use mac):
openssl dgst -md5 -binary ~/Documents/msg_Joudi.txt > ~/Documents/hash.txt
this generates hash.txt file, but its content is not in binary format, and I do not know where is the error.
Create MD5 hash of file: msgFile.txt > Convert to Binary and save:
cat msgFile.txt | openssl dgst -md5 -binary > hash.bin.txt
Save binary in Base64 format:
cat msgFile.txt | openssl dgst -md5 -binary | base64 > hash.bin.b64.txt
Save binary in Hexadecimal representation:
cat msgFile.txt | openssl dgst -md5 -binary | xxd -b > hash.bin.hex.txt
The OP says he wants binary and then says "Binary should be 0s and 1s right?".
Although, probably unlikely, if he really wants a binary number output, do this:
$ echo -e "xx\nyy" >in.txt
$ perl -ln0777e 'use Digest::MD5 "md5"; print "0b".unpack("B*",md5($_))' <in.txt
0b001111000101011000010000100101100011010101000101111000101000000011110110011010110011010000!
Meaning of above:
slurp in whole file into $_, calculate md5 digest on $_ to create binary data
use unpack to convert binary data to binary number string
If he really wants binary data, modify the above:
$ perl -ln0777e 'use Digest::MD5 "md5"; print md5 $_' <in.txt >binout.txt