I need to mimic what MySQL does when encrypting and decrypting strings using built-in functions AES_ENCRYPT() and AES_DECRYPT().
I have read a couple of blog posts and apparently MySQL uses AES 128-bit encryption for those functions. On top of that, since this encryption requires a 16-bit key, MySQL pads the string with x0 chars (\0s) until it's 16-bit in size.
The algorithm in C from MySQL source code is spotted here.
Now I need to replicate what MySQL does in a Rails application, but every single thing I tried, doesn't work.
Here's a way to replicate the behavior I am getting:
1) Create a new Rails app
rails encryption-test
cd encryption-test
2) Create a new scaffolding
script/generate scaffold user name:string password:binary
3) Edit your config/database.yml and add a test MySQL database
development:
adapter: mysql
host: localhost
database: test
user: <<user>>
password: <<password>>
4) Run the migration
rake db:migrate
5) Enter console, create an user and update its password from MySQL query
script/console
Loading development environment (Rails 2.2.2)
>> User.create(:name => "John Doe")
>> key = "82pjd12398JKBSDIGUSisahdoahOUASDHsdapdjqwjeASIduAsdh078asdASD087asdADSsdjhA7809asdajhADSs"
>> ActiveRecord::Base.connection.execute("UPDATE users SET password = AES_ENCRYPT('password', '#{key}') WHERE name='John Doe'")
That's where I got stuck. If I attempt to decrypt it, using MySQL it works:
>> loaded_user = User.find_by_sql("SELECT AES_DECRYPT(password, '#{key}') AS password FROM users WHERE id=1").first
>> loaded_user['password']
=> "password"
However if I attempt to use OpenSSL library, there's no way I can make it work:
cipher = OpenSSL::Cipher::Cipher.new("AES-128-ECB")
cipher.padding = 0
cipher.key = key
cipher.decrypt
user = User.find(1)
cipher.update(user.password) << cipher.final #=> "########gf####\027\227"
I have tried padding the key:
desired_length = 16 * ((key.length / 16) + 1)
padded_key = key + "\0" * (desired_length - key.length)
cipher = OpenSSL::Cipher::Cipher.new("AES-128-ECB")
cipher.key = key
cipher.decrypt
user = User.find(1)
cipher.update(user.password) << cipher.final #=> ""|\e\261\205:\032s\273\242\030\261\272P##"
But it really doesn't work.
Does anyone have a clue on how can I mimic the MySQL AES_ENCRYPT() and AES_DECRYPT() functions behavior in Ruby?
Thanks!
For future reference:
According to the blog post I sent before, here's how MySQL works with
the key you provide AES_ENCRYPT / DECRYPT:
"The algorithm just creates a 16 byte
buffer set to all zero, then loops
through all the characters of the
string you provide and does an
assignment with bitwise OR between the
two values. If we iterate until we
hit the end of the 16 byte buffer, we
just start over from the beginning
doing ^=. For strings shorter than 16
characters, we stop at the end of the
string."
I don't know if you can read C, but here's the mentioned snippet:
http://pastie.org/425161
Specially this part:
bzero((char*) rkey,AES_KEY_LENGTH/8); /* Set initial key */
for (ptr= rkey, sptr= key; sptr < key_end; ptr++,sptr++)
{
if (ptr == rkey_end)
ptr= rkey; /* Just loop over tmp_key until we used all key */
*ptr^= (uint8) *sptr;
}
So I came up with this method (with a help from Rob Biedenharn, from ruby forum):
def mysql_key(key)
final_key = "\0" * 16
key.length.times do |i|
final_key[i%16] ^= key[i]
end
final_key
end
That, given a string returns the key MySQL uses when encrypting and decrypting. So all you need now is:
def aes(m,k,t)
(aes = OpenSSL::Cipher::AES128.new("ECB").send(m)).key = k
aes.update(t) << aes.final
end
def encrypt(key, text)
aes(:encrypt, key, text)
end
def decrypt(key, text)
aes(:decrypt, key, text)
end
To use openssl lib, built into ruby, and then you can make the two "final" methods:
def mysql_encrypt(s, key)
encrypt(mysql_key(key), s)
end
def mysql_decrypt(s, key)
decrypt(mysql_key(key), s)
end
And you're set! Also, complete code can be found in this Gist:
http://gist.github.com/84093
:-)
Generally you don't want to pad the key, you pad/unpad the data to be encrypted/decrypted. That could be another source of problems. I suggest using test data of a complete number of blocks to eliminate this possibility.
Also, I suspect the key for the OpenSSL API requires a "literal" key, not an ASCII representation of the key as you have in your code.
Given the paucity of the OpenSSL ruby docs and if you speak a little Java, you may want to prototype in JRuby with the BouncyCastle provider - this is something that I've done to good effect when working with TwoFish (not present in OpenSSL API).
EDIT: I re-read your comment about padding the key. You have some bits/bytes confusion in your question, and I'm not sure how this applies in any case since your posted key is 89 characters (712 bits) in length. Perhaps you should try with a 128 bit key/password to eliminate this padding phenomenon?
Incidentally, MySQL devs should be spanked for weak crypto, there are better ways to stretch passwords than by simply padding with zero bytes :(
If you don't mind using an openssl implementation attr_encrypted is a gem that will allow drop-in encryption on most classes, ActiveRecord or not. It unfortunately will not be compatible with MySQL's AES_EN/DECRYPT functions though.
Related
I am using puppet to manage my MySQL deployments. Service accounts are also managed via puppet because AWS doesn't provide the ability for me to leverage OpenLDAP as an authentication provider to RDS. I was given the requirement to use the sha256_password plugin for mysql. The puppet module I am using to manage everything, https://forge.puppet.com/puppetlabs/mysql, requires me to provide a hashed password to the user resource and the function included in the module only generates the mysql_native password. How can I use Ruby to generate the required hash? Below is the code I have so far.
require 'digest'
salt = Array.new(20){rand(256).chr}.join
hash = salt + 'Password'
rounds = 5000 # 5 iteration count * 1000 multipler per mysql code
x = 0
while x < rounds do
hash = Digest::SHA2.new(256).hexdigest(hash)
x += 1
end
h_salt = [salt].pack('H*')
b_hash = [hash[0..42]].pack('m')
h_hash = [hash].pack('H*')
value = '$A$005$' + h_salt + h_hash
print value
I can't determine if I'm encoding the salt and digest correctly. I believe I have the correct format other wise.
From https://crypto.stackexchange.com/questions/77427/whats-the-algorithm-behind-mysqls-sha256-password-hashing-scheme,
the expected format:
DELIMITER[digest_type]DELIMITER[iterations]DELIMITER[salt][digest]
It seems after reviewing the source code and others implementation of it, the salt and digest are base64 encoded HEX values. Is that correct?
Some examples I leveraged were from:
https://github.com/hashcat/hashcat/issues/2305
I'm working on a Ruby project that is interacting with a webservice that I'm exchanging some encrypted data with.
I am having a very hard time decrypting something I get back from the webservice in Ruby, although in the .NET side, it's working fine, and a number of other web-based or desktop-based tools can deal with this.
The encryption method was 3DES with ECB and no padding.
Below is a test script I have been working on. I've tried everything I can think of to get these strings unpacked correctly, but to no avail.
require 'openssl'
require 'base64'
def cipher(key, encrypted)
key = key.unpack('a2'*32).map{|x| x.hex}.pack('c'*32)
encrypted = encrypted.unpack('a2'*32).map{|x| x.hex}.pack('c'*32)
OpenSSL::Cipher::ciphers.select{|c| c.include? 'des3' }.map do |cipher_name|
begin
cipher = OpenSSL::Cipher.new(cipher_name)
cipher.padding = 0
cipher.decrypt
cipher.key=key
plain = cipher.update(encrypted) + cipher.final
p "Cipher #{cipher_name} success: #{plain} #{plain.class} #{plain.length} #{plain.encoding.to_s}"
plain
rescue => e
p "Cipher #{cipher_name} failed #{e}"
nil
end
end
end
key = '202FA9B21843D7022B6466DB68327E1F'
encrypted = 'ff6f07e270ebd5c0878c67c999d87ebf'
res1 = cipher key, encrypted
key = '49CE85147B24123718AB3F4539AB1A21'
encrypted = '995604ed8016da8897f1875ebd725529'
res2 = cipher key, encrypted
p res1 == res2 ? "SUCCESS" : "FAIL"
# In both cases, the correct output should be '25588015543912470222703296730936'
A 3DES key is 24-bytes, use a full length key.
3DES uses triple encryption with essentially a 24-byte key. 202FA9B21843D7022B6466DB68327E1F is hex encoded 16-byte key.
Try repeating the first 8-bytes of the key:
202FA9B21843D7022B6466DB68327E1F202FA9B21843D702
Some 3DES implementations will repeat 8-bytes of a 16-byte key but relying on such implementation details is not a good idea.
Note: 3DES actually uses a 168-bit key because the LSb of each byte is not used. Further because there are actually three DES calls the security is only 112-bits. Additionally DES has some weak keys. There are two common modes, ede and ded, in an effort to facilitate moving from DES to 3DES thus adding more confusion.
Finally: Move from 3DES to AES in CBC mode with a random IV. Please don't continue poor security practices.
I'm Node developer, but every once in awhile I get to play around with ERB templates. I really love pulling out as much ruby as I can in these templates when I can and this idea caught my eye especially.
I have this configuration value, which should be encrypted, but is coming in plain text. The program would decrypt it like so:
var crypto = require('crypto');
var decipher = crypto.createDecipher('aes256', 'e20jhciwjf90u2r9u9ujj');
var decrypted = crypto.update('4ufujj90u19ru90u109u') + crypto.final();
I was wondering how I might go about creating an encrypted string for the above to decrypt using ruby?
So far I have:
require 'openssl'
cipher = OpenSSL::Cipher::Cipher.new('aes256');
cipher.key= 'e20jhciwjf90u2r9u9ujj'
encrypted = cipher.update('my cat is yellow and very pretty.') + cipher.final
Two problems I have:
I often get a Key length not long enough error on the ruby side.
Ruby outputs a bunch of crazy hex, whereas node seems to always take/want utf8.
Am I encrypting/decrypting safely?
Is there a way to universally translate/work laterally with these two APIs?
AES-256 uses a key of 256bits, and by default ruby uses utf-8 encoding, so each ansi character is 8bit long. So the key string must be 32 bytes.
Explicitly use an aes mode (e.g. aes-256-cbc).
Set the same iv(initial vector) on both sides
I finally succeeded with the above methods.
Here is my code:
Ruby side:
require 'openssl'
require 'base64'
cipher = OpenSSL::Cipher.new('aes-256-cbc')
cipher.encrypt
cipher.iv = 'a'*16;
cipher.key = '01234567890123456789012345678901' # should be 32 characters, 32*8=256 bits
enc = Base64.strict_encode64(cipher.update('01234567890123456789012345678901') + cipher.final)
puts enc
Javascript side:
var encrypted = new Buffer(base64Data, 'base64');
var crypto = require('crypto');
var decipher = crypto.createDecipheriv('aes-256-cbc', '01234567890123456789012345678901', 'aaaaaaaaaaaaaaaa');
var dec = decipher.update(encrypted);
console.log(Buffer.concat([dec, decipher.final()]));
I am trying to write Ruby code to check the Elliptic Curve Digital Signature Algorithm (ECDSA) signature on a particular message that I found here.
The problem is that I don't know how to convert the octet string for the public key into an OpenSSL::PKey::EC::Point object. If I were writing this in C, I would just pass the octet string to OpenSSL's o2i_ECPublicKey, which does something close to what I would want and in fact is used by the reference implementation. However, I searched the source code of Ruby (MRI) and it contains no calls to o2i_ECPublicKey so I don't know how I would use that function from Ruby without writing a C extension.
Here is the octet string, in hex. It is just a 0x04 byte followed by two 32-byte integers that represent the x and y coordinates of the point on the elliptic curve:
04fc9702847840aaf195de8442ebecedf5b095cdbb9bc716bda9110971b28a49e0ead8564ff0db22209e0374782c093bb899692d524e9d6a6956e7c5ecbcd68284
So does anyone know how to convert that string into an in OpenSSL::PKey::EC::Point in Ruby? Once I get the point object, I will use it in the following code which I believe will verify the signature:
key = OpenSSL::PKey::EC.new('secp256k1')
key.public_key = point
result = key.dsa_verify_asn1(digest, signature)
UPDATE:
Thanks to Jay-Ar Polidario I got it to work. Here is the full code I have that verifies the signature using OpenSSL. I also wrote a gem called ecdsa and I included code showing how to use my gem to do the same thing.
# coding: ASCII-8BIT
digest =
"\xbf\x91\xfb\x0b\x4f\x63\x33\x77\x4a\x02\x2b\xd3\x07\x8e\xd6\xcc" \
"\xd1\x76\xee\x31\xed\x4f\xb3\xf9\xaf\xce\xb7\x2a\x37\xe7\x87\x86"
signature_der_string =
"\x30\x45" \
"\x02\x21\x00" \
"\x83\x89\xdf\x45\xf0\x70\x3f\x39\xec\x8c\x1c\xc4\x2c\x13\x81\x0f" \
"\xfc\xae\x14\x99\x5b\xb6\x48\x34\x02\x19\xe3\x53\xb6\x3b\x53\xeb" \
"\x02\x20" \
"\x09\xec\x65\xe1\xc1\xaa\xee\xc1\xfd\x33\x4c\x6b\x68\x4b\xde\x2b" \
"\x3f\x57\x30\x60\xd5\xb7\x0c\x3a\x46\x72\x33\x26\xe4\xe8\xa4\xf1"
public_key_octet_string =
"\x04" \
"\xfc\x97\x02\x84\x78\x40\xaa\xf1\x95\xde\x84\x42\xeb\xec\xed\xf5" \
"\xb0\x95\xcd\xbb\x9b\xc7\x16\xbd\xa9\x11\x09\x71\xb2\x8a\x49\xe0" \
"\xea\xd8\x56\x4f\xf0\xdb\x22\x20\x9e\x03\x74\x78\x2c\x09\x3b\xb8" \
"\x99\x69\x2d\x52\x4e\x9d\x6a\x69\x56\xe7\xc5\xec\xbc\xd6\x82\x84"
# Verifying with openssl.
require 'openssl'
ec = OpenSSL::PKey::EC.new('secp256k1')
key_bn = OpenSSL::BN.new(public_key_octet_string, 2) # 2 means binary
ec.public_key = OpenSSL::PKey::EC::Point.new(ec.group, key_bn)
result = ec.dsa_verify_asn1(digest, signature_der_string)
puts result # => true
# Verifying with the new ECDSA gem I wrote, version 0.1.5
require 'ecdsa'
group = ECDSA::Group::Secp256k1
point = ECDSA::Format::PointOctetString.decode(public_key_octet_string, group)
signature = ECDSA::Format::SignatureDerString.decode(signature_der_string)
result = ECDSA.valid_signature?(point, digest, signature)
puts result # => true
I think it's weird that OpenSSL makes you represent the public key temporarily as a single BN (big number), because it is actually two big numbers. My gem can directly convert octet strings (as defined in the SEC2 standard) into ECDSA::Point objects.
Try the following (Tested without errors):
key = '04fc9702847840aaf195de8442ebecedf5b095cdbb9bc716bda9110971b28a49e0ead8564ff0db22209e0374782c093bb899692d524e9d6a6956e7c5ecbcd68284'
key_bn = OpenSSL::BN.new(key, 16) #Input: 16=Hexa, Output: BigNumber
group = OpenSSL::PKey::EC::Group.new('secp256k1')
point = OpenSSL::PKey::EC::Point.new(group, key_bn)
#--> <OpenSSL::PKey::EC::Point:0x5288178>
OpenSSL allows you to create a point with a group and a bignum, so I tried this:
require 'openssl'
include OpenSSL
group = PKey::EC::Group.new('secp256k1')
bignum = BN.new('04fc9702847840...')
point = PKey::EC::Point.new(group, bignum)
But it raises OpenSSL::PKey::EC::Point::Error: invalid encoding. I'm not quite sure how to troubleshoot it from here, but hopefully this helps you out a little bit.
While trying to setup an interoperable encryption system, I met a weird situation during a light "proof-of-concept".
I wrote the following code in Ruby to:
create an encrypted file from a dummy text file on my file system
decrypt the encrypted file
compare with the original file and check if they are the same
Here is the code:
require 'openssl'
require 'base64'
# Read the dummy file
data = File.read("test.txt")
# Create an encrypter
cipher = OpenSSL::Cipher::AES.new(256, :CBC)
cipher.encrypt
key = "somethingreallyreallycomplicated"
cipher.key = key
# Encrypt and save to a file
encrypted = cipher.update(data) + cipher.final
open "encrypted.txt", "w" do |io| io.write Base64.encode64(encrypted) end
# Create a decrypter
decipher = OpenSSL::Cipher::AES.new(256, :CBC)
decipher.decrypt
decipher.key = key
# Decrypt and save to a file
encrypted_data = Base64.decode64(File.read("encrypted.txt"))
plain = decipher.update(encrypted_data) + decipher.final
open "decrypted.txt", "w" do |io| io.write plain end
# Compare original message and decrypted message
puts data == plain #=> true
Everything works fine, this script outputs "true"
Then I tried to use the openssl command-line to decrypt my file with the following command:
openssl aes-256-cbc -d -a -in encrypted.txt -k somethingreallyreallycomplicated
But I got: bad magic number
How come?
You need to use the -K (upper case) and -iv options on the command line to specify key and IV explicitly as a string of hex digits. If you use -k (lower case), OpenSSL will derive key and IV from the password using a key derivation function. When OpenSSL derives a key, it will also use a "salted" ciphertext format which is incompatible with the plain blockwise CBC you are expecting.
Note that in your Ruby code, you are using the first 256 bits (32 bytes) of an ASCII string directly as a key, which is almost certainly not what you want for a real world application where security is an issue. You should use a (randomly generated) binary key, or derive a key from a password using a key derivation function such as PBKDF2, bcrypt or scrypt.