I am trying to replicate the encryption result from here in Ruby using OpenSSL: https://emvlab.org/descalc/?key=18074F7ADD44C903&iv=18074F7ADD44C903&input=4E5A56564F4C563230313641454E5300&mode=cbc&action=Encrypt&output=25C843BA5C043FFFB50F76E43A211F8D
Original string = "NZVVOLV2016AENS"
String converted to hexadecimal = "4e5a56564f4c563230313641454e53"
iv = "18074F7ADD44C903"
key = "18074F7ADD44C903"
Expected result = "9B699B4C59F1444E8D37806FA9D15F81"
Here is my ruby code:
require 'openssl'
require "base64"
include Base64
iv = "08074F7ADD44C903"
cipher = "08074F7ADD44C903"
def encode(string)
puts "Attempting encryption - Input: #{string}"
encrypt = OpenSSL::Cipher.new('DES-CBC')
encrypt.encrypt
encrypt.key = ["18074F7ADD44C903"].pack('H*') #.scan(/../).map{|b|b.hex}.pack('c*')
encrypt.iv = ["18074F7ADD44C903"].pack('H*')
result = encrypt.update(string) + encrypt.final
puts "Raw output: #{result.inspect}"
unpacked = result.unpack('H*')[0]
puts "Encrypted key is: #{unpacked}"
puts "Encrypted Should be: 9B699B4C59F1444E8D37806FA9D15F81"
return unpacked
end
res = encode("NZVVOLV2016AENS")
Output:
Encrypted key is: 9b699b4c59f1444ea723ab91e89c023a
Encrypted Should be: 9B699B4C59F1444E8D37806FA9D15F81
Interestingly, the first half of the result is correct, and the last 16 digits are incorrect.
The web site uses Zero padding by default, while the Ruby code uses PKCS#7 padding by default.
Ruby does not seem to support Zero padding, so disable the default padding and implement Zero padding yourself.
Zero padding pads to the next full block size with 0x00 values. The block size for DES is 8 bytes. If the last block of the plaintext is already filled, no padding is done:
def zeroPad(string, blocksize)
len = string.bytes.length
padLen = (blocksize - len % blocksize) % blocksize
string += "\0" * padLen
return string
end
In the encode() function (which should better be called encrypt() function) the following lines must be added before encryption:
encrypt.padding = 0 # disable PKCS#7 padding
string = zeroPad(string, 8) # enable Zero padding
The modified Ruby code then gives the same ciphertext as the web site.
Note that DES is insecure, also it' s insecure to use the key as IV (as well as a static IV). Furthermore, Zero padding is unreliable in contrast to PKCS#7 padding.
I am new to python so please forgive me if I'm asking a dumb question. In my function I generate a random byte array for a given number of bytes called "input_data", then I add bytewise some bit errors and store the result in another byte array called "output_data". The print function shows that it works exactly as expected, there are different bytes. But if I compare the byte arrays afterwards they seem to be identical!
def simulate_ber(packet_length, ber, verbose=False):
# generate input data
input_data = bytearray(random.getrandbits(8) for _ in xrange(packet_length))
if(verbose):
print(binascii.hexlify(input_data)+" <-- simulated input vector")
output_data = input_data
#add bit errors
num_errors = 0
for byte in range(len(input_data)):
error_mask = 0
for bit in range(0,7,1):
if(random.uniform(0, 1)*100 < ber):
error_mask |= 1 << bit
num_errors += 1
output_data[byte] = input_data[byte] ^ error_mask
if(verbose):
print(binascii.hexlify(output_data)+" <-- output vector")
print("number of simulated bit errors: " + str(num_errors))
if(input_data == output_data):
print ("data identical")
number of packets: 1
bytes per packet: 16
simulated bit error rate: 5
start simulation...
0d3e896d61d50645e4e3fa648346091a <-- simulated input vector
0d3e896f61d51647e4e3fe648346001a <-- output vector
number of simulated bit errors: 6
data identical
Where is the bug? I am sure the problem is somewhere between my ears...
Thank you in advance for your help!
output_data = input_data
Python is a referential language. When you do the above, both variables now refer to the same object in memory. e.g:
>>> y=['Hello']
>>> x=y
>>> x.append('World!')
>>> x
['Hello', 'World!']
>>> y
['Hello', 'World!']
Cast output_data as a new bytearray and you should be good:
output_data = bytearray(input_data)
I am trying to create a bitcoin address in ruby according to the documentation of bitcoin wiki (bitcoin creation according bitcoin wiki).
Starting point is just some random string which emulates the output of ripmed160.
Unfortunately I don't quite succeed in doing so, here is my code:
require 'base58_gmp'
tx_hash = "a8d0c0184dde994a09ec054286f1ce581bebf46446a512166eae7628734ea0a5"
ripmed160 = tx_hash[0..39]
ripmed160_with_pre = "00" + ripmed160
sha1 = Digest::SHA256.hexdigest ripmed160_with_pre
sha2 = Digest::SHA256.hexdigest sha1
bin_address = Integer("0x" + ripmed160_with_pre + sha2[0..7])
bitcoin_address = "1" + Base58GMP.encode(bin_address, 'bitcoin') # => "1GPcbTYDBwJ42MfKkedxjmJ3nrgoaNd2Sf"
I get something that looks like a bitcoin address but it is not recognised by blockchain.info so I guess it is invalid.
Can you please help me to make that work.
When you calculate the SHA256 checksum, make sure to calculate it over the actual bytes of the previous step, not the hex encoding of those bytes:
# First convert to actual bytes.
bytes = [ripmed160_with_pre].pack('H*')
# Now calculate the first hash over the raw bytes, and
# return the raw bytes again for the next hash
# (note: digest not hexdigest).
sha1 = Digest::SHA256.digest bytes
# Second SHA256, using the raw bytes from the previous step
# but this time we can use hexdigest as the rest of the code
# assumes hex encoded strings
sha2 = Digest::SHA256.hexdigest sha1
I have a byte stream that is a concatenation of sections, where each section is composed of a header plus a deflated byte stream.
I need to split this byte stream sections but the header only contains information about the data in uncompressed form, no hint about the compressed data length so I can advance properly in the stream and parse the next section.
So far the only way I found to advance past the deflated byte sequece is to parse it according to the this specification. From what I understood by reading the specification, a deflate stream is composed of blocks, which can be compressed blocks or literal blocks.
Literal blocks contain a size header which can be used to easily advance past it.
Compressed blocks are composed with 'prefix codes', which are bit sequences of variable length that have special meanings to the deflate algorithm. Since I'm only interested in finding out the deflated stream length, I guess the only code I need to look for is '0000000' which according to the specification signals the end of block.
So I came up with this coffeescript function to parse the deflate stream(I'm working on node.js)
# The job of this function is to return the position
# after the deflate stream contained in 'buffer'. The
# deflated stream begins at 'pos'.
advanceDeflateStream = (buffer, pos) ->
byteOffset = 0
finalBlock = false
while 1
if byteOffset == 6
firstTypeBit = 0b00000001 & buffer[pos]
pos++
secondTypeBit = 0b10000000 & buffer[pos]
type = firstTypeBit | (secondTypeBit << 1)
else
if byteOffset == 7
pos++
type = buffer[pos] & (0b01100000 >>> byteOffset)
if type == 0
# Literal block
# ignore the remaining bits and advance position
byteOffset = 0
pos++
len = buffer.readUInt16LE(pos)
pos += 2
lenComplement = buffer.readUInt16LE(pos)
if (len ^ ~lenComplement)
throw new Error('Literal block lengh check fail')
pos += (2 + len) # Advance past literal block
else if type in [1, 2]
# huffman block
# we are only interested in finding the 'block end' marker
# which is signaled by the bit string 0000000 (256)
eob = false
matchedZeros = 0
while !eob
byte = buffer[pos]
for i in [byteOffset..7]
# loop the remaining bits looking for 7 consecutive zeros
if (byte ^ (0b10000000 >>> byteOffset)) >>> (7 - byteOffset)
matchedZeros++
else
# reset counter
matchedZeros = 0
if matchedZeros == 7
eob = true
break
byteOffset++
if !eob
byteOffset = 0
pos++
else
throw new Error('Invalid deflate block')
finalBlock = buffer[pos] & (0b10000000 >>> byteOffset)
if finalBlock
break
return pos
To check if this works, I wrote a simple mocha test case:
zlib = require 'zlib'
test 'sample deflate stream', (done) ->
data = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' # length 30
zlib.deflate data, (err, deflated) ->
# deflated.length == 11
advanceDeflateStream(deflated, 0).shoudl.eql(11)
done()
The problem is that this test fails and I do not know how to debug it. I accept any answer that points what I missed in the parsing algorithm or contains a correct version of the above function in any language.
The only way to find the end of a deflate stream or even a deflate block is to decode all of the Huffman codes contained within. There is no bit pattern that you can search for that can not appear earlier in the stream.
I'm trying to get a ruby implementation of an encryption lib that's apparently popular in the Java world -- PBEWithMD5AndDES
Does anyone know how to use openssl or another open source gem to perform encryption/decryption that's compatible with this format?
Updated:
I used a gem chilkat to implement it but it is paid, i need an opensource solution.
I know it is super old but I had the same problem and just solved it so here it goes
to encrypt, where salt is your salt sting, passkey is your password key string and iterations is number of iterations you want to use
def encrypt_account_number
cipher = OpenSSL::Cipher::Cipher.new("DES")
cipher.encrypt
cipher.pkcs5_keyivgen passkey, salt,iterations,digest
encrypted_account_number = cipher.update(account_number)
encrypted_account_number << cipher.final
Base64.encode64(encrypted_account_number )
end
def decrypt_account_number
cipher = OpenSSL::Cipher::Cipher.new("DES")
base_64_code = Base64.decode64(account_number)
cipher.decrypt
cipher.pkcs5_keyivgen passkey, salt,iterations,digest
decrypted_account_number = cipher.update base_64_code
decrypted_account_number << cipher.final
decrypted_account_number
end
You don't need to actually implement PBEWithMD5andDES assuming ruby has a DES implementation. What you need to implement is the key derivation function ( who you get a key out of a password) and then feed that derived key to DES with the appropriate mode and padding.
Thankfully, the key derivation function is not particularly security critical in implementation, so you can do it yourself safely enough. According to the rfc, PBEwithMD5AndDES is actually the PBKDF1 ( a ker derivation function) used with DES in CBC mode .
PBKDF1 does not look that hard to implement . Looks like you can do it with a for loop and an md5 call.
Note that you may still get some odd results because of the possibility of a different padding scheme being used in Java and Ruby. I assume that the spec one is pkcs 1.5 padding, but at a quick glance, I can't confirm this
5.1 PBKDF1
PBKDF1 applies a hash function, which shall be MD2 [6], MD5 [19] or
SHA-1 [18], to derive keys. The length of the derived key is bounded
by the length of the hash function output, which is 16 octets for MD2
and MD5 and 20 octets for SHA-1. PBKDF1 is compatible with the key
derivation process in PKCS #5 v1.5.
PBKDF1 is recommended only for compatibility with existing
applications since the keys it produces may not be large enough for
some applications.
PBKDF1 (P, S, c, dkLen)
Options: Hash underlying hash function
Input: P password, an octet string
S salt, an eight-octet string
c iteration count, a positive integer
dkLen intended length in octets of derived key,
a positive integer, at most 16 for MD2 or
MD5 and 20 for SHA-1
Output: DK derived key, a dkLen-octet string
Steps:
1. If dkLen > 16 for MD2 and MD5, or dkLen > 20 for SHA-1, output
"derived key too long" and stop.
2. Apply the underlying hash function Hash for c iterations to the
concatenation of the password P and the salt S, then extract
the first dkLen octets to produce a derived key DK:
T_1 = Hash (P || S) ,
T_2 = Hash (T_1) ,
...
T_c = Hash (T_{c-1}) ,
DK = Tc<0..dkLen-1>
3. Output the derived key DK.
For what its' worth, I'm posting my python code, which actually works (I have tons of encrypted values which were done using org.jasypt.util.text.BasicTextEncryptor, I needed to decrypt them.)
import base64
import hashlib
from Crypto.Cipher import DES
"""
Note about PBEWithMD5AndDES in java crypto library:
Encrypt:
Generate a salt (random): 8 bytes
<start derived key generation>
Append salt to the password
MD5 Hash it, and hash the result, hash the result ... 1000 times
MD5 always gives us a 16 byte hash
Final result: first 8 bytes is the "key" and the next is the "initialization vector"
(there is something about the first 8 bytes needing to be of odd paraity, therefore
the least significant bit needs to be changed to 1 if required. We don't do it,
maybe the python crypto library does it for us)
<end derived key generation>
Pad the input string with 1-8 bytes (note: not 0-7, so we always have padding)
so that the result is a multiple of 8 bytes. Padding byte value is same as number of
bytes being padded, eg, \x07 if 7 bytes need to be padded.
Use the key and iv to encrypt the input string, using DES with CBC mode.
Prepend the encrypted value with the salt (needed for decrypting since it is random)
Base64 encode it -> this is your result
Decrypt:
Base64 decode the input message
Extract the salt (first 8 bytes). The rest is the encoded text.
Use derived key generation as in Encrypt above to get the key and iv
Decrypt the encoded text using key and iv
Remove padding -> this is your result
(I only have implemented decrypt here since that's all I needed,
but encrypt should be straighforward as well)
"""
def get_derived_key(password, salt, count):
key = password + salt
for i in range(count):
m = hashlib.md5(key)
key = m.digest()
return (key[:8], key[8:])
def decrypt(msg, password):
msg_bytes = base64.b64decode(msg)
salt = msg_bytes[:8]
enc_text = msg_bytes[8:]
(dk, iv) = get_derived_key(password, salt, 1000)
crypter = DES.new(dk, DES.MODE_CBC, iv)
text = crypter.decrypt(enc_text)
# remove the padding at the end, if any
return re.sub(r'[\x01-\x08]','',text)
I've updated python script from user3392439, with encrypt support. Wish it helpful.
import base64
import hashlib
import re
import os
from Crypto.Cipher import DES
"""
Note about PBEWithMD5AndDES in java crypto library:
Encrypt:
Generate a salt (random): 8 bytes
<start derived key generation>
Append salt to the password
MD5 Hash it, and hash the result, hash the result ... 1000 times
MD5 always gives us a 16 byte hash
Final result: first 8 bytes is the "key" and the next is the "initialization vector"
(there is something about the first 8 bytes needing to be of odd paraity, therefore
the least significant bit needs to be changed to 1 if required. We don't do it,
maybe the python crypto library does it for us)
<end derived key generation>
Pad the input string with 1-8 bytes (note: not 0-7, so we always have padding)
so that the result is a multiple of 8 bytes. Padding byte value is same as number of
bytes being padded, eg, \x07 if 7 bytes need to be padded.
Use the key and iv to encrypt the input string, using DES with CBC mode.
Prepend the encrypted value with the salt (needed for decrypting since it is random)
Base64 encode it -> this is your result
Decrypt:
Base64 decode the input message
Extract the salt (first 8 bytes). The rest is the encoded text.
Use derived key generation as in Encrypt above to get the key and iv
Decrypt the encoded text using key and iv
Remove padding -> this is your result
(I only have implemented decrypt here since that's all I needed,
but encrypt should be straighforward as well)
"""
def get_derived_key(password, salt, count):
key = password + salt
for i in range(count):
m = hashlib.md5(key)
key = m.digest()
return (key[:8], key[8:])
def decrypt(msg, password):
msg_bytes = base64.b64decode(msg)
salt = msg_bytes[:8]
enc_text = msg_bytes[8:]
(dk, iv) = get_derived_key(password, salt, 1000)
crypter = DES.new(dk, DES.MODE_CBC, iv)
text = crypter.decrypt(enc_text)
# remove the padding at the end, if any
return re.sub(r'[\x01-\x08]','',text)
def encrypt(msg, password):
salt = os.urandom(8)
pad_num = 8 - (len(msg) % 8)
for i in range(pad_num):
msg += chr(pad_num)
(dk, iv) = get_derived_key(password, salt, 1000)
crypter = DES.new(dk, DES.MODE_CBC, iv)
enc_text = crypter.encrypt(msg)
return base64.b64encode(salt + enc_text)
def main():
msg = "hello, world"
passwd = "mypassword"
s = encrypt(msg, passwd)
print s
print decrypt(s, passwd)
if __name__ == "__main__":
main()
For #cooljohny
I do not really recall how this code works, but I'm 99% sure it does. I wrote it by carefully digging through the spec from the Java implementation of pbewithmd5anddes, and testing this python against that. I delivered exactly this code to a client, and it worked fine for them. I changed the constants before pasting it here, but that's all. You should be able to confirm that it produces the same encrypted output as the Java lib, and then replicate it in ruby. Good luck!
import base64
from Crypto.Cipher import DES
from passlib.utils.pbkdf2 import pbkdf1
password = 'xxxxxxx'
iterations = 22
salt_bytes = [19,15,78,45,34,90,12,11]
# convert saltBytes to a string
salt_string = ''.join([chr(a) for a in salt_bytes])
# a sample request
raw_data = '''{"something":"to","encrypt":"here"}'''
# from the standard...
padding_value = (8 - (raw_data.__len__() % 8))
padding_data = chr(padding_value) * padding_value
padded_data = raw_data + padding_data
# 22 iterations, 16 is the # of bytes in an md5 digest
pbkres = pbkdf1(password, salt_string, iterations, 16, 'md5')
# split the digest into two 8-byte halves
# this gives the DES secret key and initializing vector
des_key, iv = pbkres[0:8], pbkres[8:16]
# encrypt with DES
cipher = DES.new(des_key, DES.MODE_CBC, iv)
cmsg = cipher.encrypt(padded_data)
# and base64 encode
base64.b64encode(cmsg)