HMAC SHA256 hash generated changes when variable is used in the hashing function than using the literal.
I have to concatenate 4 parameters to generate a message string that is hashed using secret key. The concatenated message string generates a different hash than using the value of message as a literal.
require 'base64'
require 'openssl'
securityKey = 'A93reRTUJHsCuQSHR+L3GxqOJyDmQpCgps102ciuabc='
content = 'hello'
id = '1000000855'
tsp = '1460852115'
guid = '75c6016eaa1e43b4807ba25232797714'
contentmd5 = Base64.encode64(OpenSSL::Digest::MD5.digest(content))
inputString = id + tsp + guid + contentmd5
puts inputString
#Input String is
#'1000000855146085211575c6016eaa1e43b4807ba25232797714XUFAKrxLKna5cZ2REBfFkg=='
digest = OpenSSL::Digest.new('sha256')
hmac = OpenSSL::HMAC.digest(digest, securityKey, inputString)
securityToken = Base64.encode64(hmac)
puts securityToken
#Hash generated is 7ihOEZNeoJMwjLt84I8WfN5b0VwgYNOg8abPA3nZ0SM=
digest = OpenSSL::Digest.new('sha256')
hmac = OpenSSL::HMAC.digest(digest, securityKey, '1000000855146085211575c6016eaa1e43b4807ba25232797714XUFAKrxLKna5cZ2REBfFkg==')
securityToken = Base64.encode64(hmac)
puts securityToken
#Hash generated is gPNytNGMbhg8b27rklqmEK/9xjNAcOq+7nldzyDL4g0=
looks like Base64.encode64 appends a "\n" to the end of its output so
from docs
encode64(bin) Returns the Base64-encoded
version of bin. This method complies with RFC 2045. Line feeds are
added to every 60 encoded characters.
this
contentmd5 = Base64.encode64(OpenSSL::Digest::MD5.digest(content))
returns
"XUFAKrxLKna5cZ2REBfFkg==\n"
not
"XUFAKrxLKna5cZ2REBfFkg=="
--
you can use strict_encode64 to not include line feeds so:
contentmd5 = Base64.strict_encode64(OpenSSL::Digest::MD5.digest(content))
returns
=> "XUFAKrxLKna5cZ2REBfFkg=="
Related
I am using Ruby's Open SSL bindings to do AES-256 encryption. I can encrypt a non-empty string. However, when attempting to encrypt an empty string, Ruby raises an exception complaining that the data must not be empty. How can I encrypt an empty string using Ruby's OpenSSL bindings?
Code to reproduce the problem
require "openssl"
KEY = OpenSSL::Cipher::Cipher.new("aes-256-cbc").random_key
def encrypt(plaintext)
cipher = OpenSSL::Cipher::Cipher.new("aes-256-cbc")
cipher.encrypt
iv = cipher.random_iv
cipher.iv = iv
cipher.key = KEY
ciphertext = cipher.update(plaintext) # <- ArgumentError here
ciphertext << cipher.final
[iv, ciphertext]
end
def decrypt(iv, ciphertext)
cipher = OpenSSL::Cipher::Cipher.new("aes-256-cbc")
cipher.decrypt
cipher.iv = iv
cipher.key = KEY
plaintext = cipher.update(ciphertext)
plaintext << cipher.final
plaintext
end
p decrypt(*encrypt("foo")) # "foo"
p decrypt(*encrypt(""))
# /tmp/foo.rb:11:in `update': data must not be empty (ArgumentError)
# from /tmp/foo.rb:11:in `encrypt'
# from /tmp/foo.rb:27:in `<main>'
Versions
ruby-2.2.2p95
OpenSSL::VERSION is "1.1.0"
Microsoft SQL Server 2014 (12.0.2000.8)
Why do I want to encrypt empty strings?
I am writing an ETL program to migrate data from one database to a SqlServer database. Certain columns from the source database must be encrypted before writing them to the destination database. The source columns may contain any data, including empty strings. The destination columns are usually non-nullable. The destination columns will be decrypted by .net code.
Goal #1: No information about the encrypted field, including whether or not it even exists, should be recoverable without properly decrypting it. An encrypted empty string should be indistinguishable from any other encrypted data.
Goal #2: The .net code that will decrypt these values should not need to handle empty strings specially.
If I can get openssl to encrypt empty strings, I will achieve both of these goals.
Workaround - Don't encrypt empty strings
I could just not encrypt empty strings, passing them through.
def encrypt(plaintext)
return plaintext if plaintext.empty?
...
end
def decrypt(iv, ciphertext)
return ciphertext if ciphertext.empty?
...
end
This has the disadvantages of exposing information, and also of requiring cooperating code to be written on the .net side.
Workaround - Add some constant to the plaintext
I could add some constant string to the plaintext before encryption, and remove it after decryption:
PLAINTEXT_SUFFIX = " "
def encrypt(plaintext)
plaintext += PLAINTEXT_SUFFIX
...
end
def decrypt(iv, ciphertext)
...
plaintext.chomp(PLAINTEXT_SUFFIX)
end
This hides whether the data exists or not, but still requires cooperating .net code.
As suggested by #ArtjomB, it's as simple as not calling Cipher#update with the empty string. The value returned by Cipher#final then properly encrypts an empty string.
require "openssl"
KEY = OpenSSL::Cipher::Cipher.new("aes-256-cbc").random_key
def encrypt(plaintext)
cipher = OpenSSL::Cipher::Cipher.new("aes-256-cbc")
cipher.encrypt
iv = cipher.random_iv
cipher.iv = iv
cipher.key = KEY
ciphertext = ""
ciphertext << cipher.update(plaintext) unless plaintext.empty?
ciphertext << cipher.final
[iv, ciphertext]
end
def decrypt(iv, ciphertext)
cipher = OpenSSL::Cipher::Cipher.new("aes-256-cbc")
cipher.decrypt
cipher.iv = iv
cipher.key = KEY
plaintext = cipher.update(ciphertext)
plaintext << cipher.final
end
p decrypt(*encrypt("foo")) # "foo"
p decrypt(*encrypt("")) # ""
If you can use DBMS provided encryption functions, then, MySQL AES_ENCRYPT, seems to be able to encrypt blank string.
For example:
UPDATE some_table
SET some_column = AES_ENCRYPT('',UNHEX('F3229A0B371ED2D9441B830D21A390C3'));
It's AES-128 by default, I am guessing that will be a problem as you need AES-256. Also, not sure which DBMS you are using and whether that DBMS has encryption functions.
I have two scripts for encrypting and decrypting messages using AES.
Here's encrypt.rb:
require 'openssl'
require 'base64'
require 'digest'
KEY = 'sekrit_key'
MESSAGE = "My Name is Rabbit and I don't know anything!"
cipher = OpenSSL::Cipher::AES256.new(:CFB)
cipher.encrypt
cipher.key = KEY
# hexdigest the IV to make sure encode64 doesn't fuck up
iv = Digest::MD5.hexdigest(cipher.random_iv)
cipher.iv = iv
encrypted = cipher.update(secret_message)+cipher.final
puts Base64.urlsafe_encode64(iv+'|'+encrypted.encode)
... and decrypt.rb:
require 'openssl'
require 'base64'
KEY = 'sekrit_key'
encrypted_message = STDIN.read.strip
parts = Base64.urlsafe_decode64(encrypted_message ).split('|')
iv = parts[0]
encrypted = parts[1]
decipher = OpenSSL::Cipher::AES256.new(:CFB)
decipher.decrypt
decipher.key = KEY
decipher.iv = iv
message = decipher.update(encrypted)+decipher.final
if message.eql?("My Name is Rabbit and I don't know anything!")
print '.'
else
puts
puts encrypted_message
puts message
end
When I now continuosly run both scripts, My output is cut off quite often!
$ while true; do ruby encrypt.rb | ruby decrypt.rb; done
....
YmZhNDg2ODJjNGZiOGIzZTcyMzAwYzMxZWUwNWI0Y2V8w2aJk930EL3gh3rfQsd2B3xZKy5wjoCzlZoYHBgmv6m51ZwAWQHGtCJoNRg=
My Name is Rabbi
..
YjMwNDQxOGRjMjg4NGEzOThmM2IwNGFiZDBiZTQxZGZ8OfLyjGQGKV3PPUpCvfL08IDuk7M7d3w7fj6F5Rql94jkRdwaCuuMfedqtFk=
My Name is Rabbit and
..
OWUxYzFlZWU5MTc4NGZjYWYxYzZiOGEwOTBjOGMxYzJ8g7I4X_Dt6K9ufByMhLBGlpoYCv8vlR0lTBqP-zS647tmmFh81rXdR8T-UkM=
My Name i
....
# and so on
Why are so many of my messages cut off?
Update: When using a fixed IV in encrypt.rb (eg. substitute cipher.random_iv with KEY) instead of a randomly generated, the problem does not occur.
The problem stems from the fact that you're treating binary data as strings. iv as well as encrypted.encode are binary and you're concatenating them with "|" (a string). Both iv as well as the message may contain the pipe character which will cause problems when splitting. In general, it's best to base64 both parts separately.
Here's working code:
encrypt.rb
require 'openssl'
require 'base64'
require 'digest'
KEY = 'sekrit_key123456' * 2
MESSAGE = "My Name is Rabbit and I don't know anything!"
cipher = OpenSSL::Cipher::AES256.new(:CFB)
cipher.encrypt
cipher.key = KEY
iv = cipher.random_iv
encrypted = cipher.update(MESSAGE)+cipher.final
puts Base64.strict_encode64(iv)+'|'+Base64.strict_encode64(encrypted.encode)
decrypt.rb
require 'openssl'
require 'base64'
KEY = 'sekrit_key123456' * 2 # key needs to be the right length
encrypted_message = STDIN.read.strip
parts = encrypted_message.split('|')
iv = Base64.strict_decode64(parts[0])
encrypted = Base64.strict_decode64(parts[1])
decipher = OpenSSL::Cipher::AES256.new(:CFB)
decipher.decrypt
decipher.key = KEY
decipher.iv = iv
message = decipher.update(encrypted)+decipher.final
if message.eql?("My Name is Rabbit and I don't know anything!")
print '.'
else
puts
puts encrypted_message
puts message
end
Also notice that random_iv already assigns iv to cipher, so you don't need to (see source under http://apidock.com/ruby/v1_9_3_125/OpenSSL/Cipher/random_iv ).
I have a request that looks like this ;
url = "https://api-3t.sandbox.paypal.com/nvp?METHOD=DoExpressCheckoutPayment&TOKEN=#{transaction.token}
&PAYERID=#{transaction.payer_id}&PAYMENTREQUEST_n_PAYMENTACTION=sale"
url = CGI.escape(url)
uri = URI(url)
res = Net::HTTP.get_response(uri)
and res.body looks like this; TOKEN=EC%2d7UM71457T34680821&TIMESTAMP=2013%2d11%2d03T21%3a19%3a11Z&CORRELATIONID=3b73c396244ff&ACK=Success&VERSION=98&BUILD=8334781
How can i get the TOKEN and ACK values from the string? am not sure params works here?
Any ideas?
The body is URI-encoded, just like GET (or some POST) params. You could unpack it manually, by doing something like this:
require 'uri'
# body takes place of res.body for this example
body = 'TOKEN=EC%2d7UM71457T34680821&TIMESTAMP=2013%2d11%2d03' +
'T21%3a19%3a11Z&CORRELATIONID=3b73c396244ff&AC' +
'K=Success&VERSION=98&BUILD=8334781'
# First split into key/value pairs, and use inject to start building a hash
results = body.split('&').inject( {} ) do |hash,kv|
# Split each key/value pair
k,v = kv.split('=').map do |uri_encoded_value|
# Decode - innermost loop, because it may contain encoded '&' or '='
URI.decode(uri_encoded_value)
end
# Add to hash we are building with inject
hash[k] = v
hash
end
=> {"TOKEN"=>"EC-7UM71457T34680821", "TIMESTAMP"=>"2013-11-03T21:19:11Z",
"CORRELATIONID"=>"3b73c396244ff", "ACK"=>"Success", "VERSION"=>"98",
"BUILD"=>"8334781"}
Actually though URI can do nearly all of this for you (and deal with variations in the format better than above), with the decode_www_form class method.
params = {}
URI.decode_www_form( body ).each do |k,v|
params[k] = v
end
params
=> {"TOKEN"=>"EC-7UM71457T34680821", "TIMESTAMP"=>"2013-11-03T21:19:11Z",
"CORRELATIONID"=>"3b73c396244ff", "ACK"=>"Success", "VERSION"=>"98",
"BUILD"=>"8334781"}
So I have a secure messaging script written up that uses JSON to send messages between a client and server. At one point I'm generating signed message digests using SHA1 and this is giving me some characters that I can't parse in JSON.generate to send my messages in JSON format around. Can anyone help me get around this error "in `encode': "\xAB" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError)" ?
The main problem I'm having is with this section of code:
#make new private / public rsa key-pair
rsakey = OpenSSL::PKey::RSA.new 2048
#hash the key using sha1
sha1 = OpenSSL::Digest::SHA1.new
digest = sha1.digest(rsakey.public_key.to_pem)
pubkey = JSON.generate({
key: rsakey.public_key.to_pem,
digest: digest
})
It's not allowing me to do a JSON.generate of digest. Does anyone know of a workaround or another way to encode my message digest?
My full code is below:
# encoding: utf-8
require 'socket'
require 'openssl'
require 'json'
port = 9090
s = TCPServer.open(port)
#make new private / public rsa key-pair
rsakey = OpenSSL::PKey::RSA.new 2048
#hash the key using sha1
sha1 = OpenSSL::Digest::SHA1.new
digest = sha1.digest(rsakey.public_key.to_pem)
pubkey = JSON.generate({
key: rsakey.public_key.to_pem,
digest: digest
})
loop {
client = s.accept
#get public key from alice
incoming = client.gets()
alice = JSON.parse(incoming)
alice_key = OpenSSL::PKey::RSA.new alice['key']
#send public key to alice
puts pubkey
client.puts pubkey
#get encrypted package from alice
json_full_package = client.gets()
full_package = JSON.parse(json_full_package)
#decrypt and print package
cipher = OpenSSL::Cipher.new("DES3")
cipher.decrypt
key = rsakey.private_decrypt(full_package['key'])
iv = rsakey.private_decrypt(full_package['iv'])
json_package = cipher.update(full_package['package'])
package = JSON.parse(json_package)
decrypted_digest = alice_key.public_decrypt(package['signed_digest'])
sha1 = OpenSSL::Digest::SHA1.new
digest = sha1.digest(package['data'])
throw 'failed digest' unless digest == decrypted_digest
puts package['data']
client.close
}
# encoding: utf-8
require 'socket'
require 'openssl'
require 'json'
host = 'lab1-15.eng.utah.edu'
port = 9090
s = TCPSocket.open(host, port)
pubkey_q = false
keyF = File.new("public_key.pem", 'w')
#generate alice's key pair
key = OpenSSL::PKey::RSA.new 2048
to_bob_public = JSON.generate({
key: key.public_key.to_pem
})
s.send(to_bob_public)
#get public key certificate from bob
while line = s.gets
puts line.chop
bob = JSON.parse(line)
end
bob_key = OpenSSL::PKey::RSA.new bob['key']
bob_digest = bob['digest']
#verify public key
sha1 = OpenSSL::Digest::SHA1.new
t_digest = sha1.digest(bob['key'])
throw "not verified" unless t_digest == bob_digest
data = File.read('document') #data is original message
#hash the document using sha1
sha1 = OpenSSL::Digest::SHA1.new
digest = sha1.digest(data)
#sign with private key
signed_digest = key.private_encrypt(digest)
#package this in json
package = JSON.generate({
signed_digest: signed_digest,
data: data
})
#make cipher for encryption
cipher = OpenSSL::Cipher.new("DES3")
cipher.encrypt
key = cipher.random_key
iv = cipher.random_iv
#encrypt data
encrypted = cipher.update(package)
#encrypt key and iv using bob's public key
encrypted_cipher_key = bob_key.public_encrypt(key)
encrypted_cipher_iv = bob_key.public_encrypt(iv)
full_package = JSON.generate({
key: encrypted_cipher_key,
iv: encrypted_cipher_iv,
package: encrypted
})
#send full_package to bob
s.send(full_package)
s.close
Use #hexdigest instead of #digest and you'll get hex. Right now you're attempting to encode binary into JSON (as that's what #digest returns) and that is causing the encode failure.
I cannot get Perl and Ruby to agree on CBC AES:
Perl
use Crypt::CBC;
use MIME::Base64::Perl;
my $cipher = Crypt::CBC->new(
-key => 'd2cb415e067c7b13',
-iv => 'e36dc751d0433f05', #random 16chars!!!!!! shold NOT repeat between requests
-cipher => 'OpenSSL::AES', #this is same as Rijndael
-literal_key => 1,
-header => "none",
-keysize => 16
);
$encypted = $cipher->encrypt("a really really long long text has differrent results???");
$base64 = encode_base64($encypted);
print("Ciphertext(b64): $base64");
$de_base64 = decode_base64($base64);
$decrypted = $cipher->decrypt($de_base64);
$c = $cipher->finish;
Ciphertext(b64): qz4eSQaFkQUkDOyJSbZf5W03HoldwtgvTLq0yJFRViKJnytf3PVSCGW2CYDjO+tRqV20oxeB2VPa
7NqN1TDSNQ==
there's a newline after the 2VPa section and another at the end
Ruby
require 'openssl'
require 'digest/sha2'
require 'base64'
message = "a really really long long text has differrent results???"
cipher = OpenSSL::Cipher.new('aes-128-cbc')
# digest the key, iv and hmac_key so we have 16-byte length
# also, it looks more of a funky password
# prepare cipher
cipher.encrypt
cipher.key = aes_key = "d2cb415e067c7b13"
cipher.iv = aes_iv = "e36dc751d0433f05"
encrypted = cipher.update(message)
encrypted << cipher.final()
b64_encoded = Base64.encode64(encrypted).encode('utf-8') #strict_encode64 guarantees no newlines, encode64 is default
puts "AES Key : '#{aes_key}'"
puts "AES IV : '#{aes_iv}'"
puts "Ciphertext(b64): '#{b64_encoded}'"
Ciphertext(b64): 'qz4eSQaFkQUkDOyJSbZf5W03HoldwtgvTLq0yJFRViKJnytf3PVSCGW2CYDj
O+tRqV20oxeB2VPa7NqN1TDSNQ==
'
Note the newlines chars after CYDj and after ==
I've seen the following: Perl & Ruby exchange AES encrypted information, but I'm not using padding=0
Newlines are not significant in Base64. You got exactly the same result from both languages.
While there should be absolutely no reason to do so, you could make the Perl version return the same string as the Ruby version as follows:
$base64 = encode_base64($encypted, '');
$base64 =~ s/\G.{60}\K/\n/sg;
The encode_base64 function takes a second parameter, called "eol" (end of line) which, by default, is '\n'.
The returned encoded string is broken into lines of no more than 76
characters each and it will end with $eol unless it is empty
Try:
$base64 = encode_base64($encypted, '');
instead.