How to unpack 7-bits at a time in ruby? - ruby

I'm trying to format a UUIDv4 into a url friendly string. The typical format in base16 is pretty long and has dashes:
xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx
To avoid dashes and underscores I was going to use base58 (like bitcoin does) so each character fully encode sqrt(58).floor = 7 bits.
I can pack the uuid into binary with:
[ uuid.delete('-') ].pack('H*')
To get 8-bit unsigned integers its:
binary.unpack('C*')
How can i unpack every 7-bits into 8-bit unsigned integers? Is there a pattern to scan 7-bits at a time and set the high bit to 0?

require 'base58'
uuid ="123e4567-e89b-12d3-a456-426655440000"
Base58.encode(uuid.delete('-').to_i(16))
=> "3fEgj34VWmVufdDD1fE1Su"
and back again
Base58.decode("3fEgj34VWmVufdDD1fE1Su").to_s(16)
=> "123e4567e89b12d3a456426655440000"
A handy pattern to reconstruct the uuid format from a template
template = 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx'
src = "123e4567e89b12d3a456426655440000".each_char
template.each_char.reduce(''){|acc, e| acc += e=='-' ? e : src.next}
=> "123e4567-e89b-12d3-a456-426655440000"

John La Rooy's answer is great, but I just wanted to point out how simple the Base58 algorithm is because I think it's neat. (Loosely based on the base58 gem, plus bonus original int_to_uuid function):
ALPHABET = "123456789abcdefghijkmnopqrstuvwxyzABCDEFGHJKLMNPQRSTUVWXYZ".chars
BASE = ALPHABET.size
def base58_to_int(base58_val)
base58_val.chars
.reverse_each.with_index
.reduce(0) do |int_val, (char, index)|
int_val + ALPHABET.index(char) * BASE ** index
end
end
def int_to_base58(int_val)
''.tap do |base58_val|
while int_val > 0
int_val, mod = int_val.divmod(BASE)
base58_val.prepend ALPHABET[mod]
end
end
end
def int_to_uuid(int_val)
base16_val = int_val.to_s(16)
[ 8, 4, 4, 4, 12 ].map do |n|
base16_val.slice!(0...n)
end.join('-')
end
uuid = "123e4567-e89b-12d3-a456-426655440000"
int_val = uuid.delete('-').to_i(16)
base58_val = int_to_base58(int_val)
int_val2 = base58_to_int(base58_val)
uuid2 = int_to_uuid(int_val2)
printf <<END, uuid, int_val, base_58_val, int_val2, uuid2
Input UUID: %s
Input UUID as integer: %d
Integer encoded as base 58: %s
Integer decoded from base 58: %d
Decoded integer as UUID: %s
END
Output:
Input UUID: 123e4567-e89b-12d3-a456-426655440000
Input UUID as integer: 24249434048109030647017182302883282944
Integer encoded as base 58: 3fEgj34VWmVufdDD1fE1Su
Integer decoded from base 58: 24249434048109030647017182302883282944
Decoded integer as UUID: 123e4567-e89b-12d3-a456-426655440000

Related

Ruby openssl encryption with DES-CBC incorrect result

I am trying to replicate the encryption result from here in Ruby using OpenSSL: https://emvlab.org/descalc/?key=18074F7ADD44C903&iv=18074F7ADD44C903&input=4E5A56564F4C563230313641454E5300&mode=cbc&action=Encrypt&output=25C843BA5C043FFFB50F76E43A211F8D
Original string = "NZVVOLV2016AENS"
String converted to hexadecimal = "4e5a56564f4c563230313641454e53"
iv = "18074F7ADD44C903"
key = "18074F7ADD44C903"
Expected result = "9B699B4C59F1444E8D37806FA9D15F81"
Here is my ruby code:
require 'openssl'
require "base64"
include Base64
iv = "08074F7ADD44C903"
cipher = "08074F7ADD44C903"
def encode(string)
puts "Attempting encryption - Input: #{string}"
encrypt = OpenSSL::Cipher.new('DES-CBC')
encrypt.encrypt
encrypt.key = ["18074F7ADD44C903"].pack('H*') #.scan(/../).map{|b|b.hex}.pack('c*')
encrypt.iv = ["18074F7ADD44C903"].pack('H*')
result = encrypt.update(string) + encrypt.final
puts "Raw output: #{result.inspect}"
unpacked = result.unpack('H*')[0]
puts "Encrypted key is: #{unpacked}"
puts "Encrypted Should be: 9B699B4C59F1444E8D37806FA9D15F81"
return unpacked
end
res = encode("NZVVOLV2016AENS")
Output:
Encrypted key is: 9b699b4c59f1444ea723ab91e89c023a
Encrypted Should be: 9B699B4C59F1444E8D37806FA9D15F81
Interestingly, the first half of the result is correct, and the last 16 digits are incorrect.
The web site uses Zero padding by default, while the Ruby code uses PKCS#7 padding by default.
Ruby does not seem to support Zero padding, so disable the default padding and implement Zero padding yourself.
Zero padding pads to the next full block size with 0x00 values. The block size for DES is 8 bytes. If the last block of the plaintext is already filled, no padding is done:
def zeroPad(string, blocksize)
len = string.bytes.length
padLen = (blocksize - len % blocksize) % blocksize
string += "\0" * padLen
return string
end
In the encode() function (which should better be called encrypt() function) the following lines must be added before encryption:
encrypt.padding = 0 # disable PKCS#7 padding
string = zeroPad(string, 8) # enable Zero padding
The modified Ruby code then gives the same ciphertext as the web site.
Note that DES is insecure, also it' s insecure to use the key as IV (as well as a static IV). Furthermore, Zero padding is unreliable in contrast to PKCS#7 padding.

Implement a basic datatype in BinData

I am trying to implement binary16 encoding for a half-precision floating point type.
The code is working except for one detail: It returns an object with the three properties (sign, exponent, fraction), but I would like it to return the float. Right now, I have to call to_f to get to the float. I would like this to work the way the integrated int and float classes work.
Here's my code:
require 'bindata'
class Binary16Be < BinData::Record
# naming based on https://en.wikipedia.org/wiki/Half-precision_floating-point_format
bit1 :sign_bit
bit5 :exponent
bit10 :fraction
def sign
sign_bit.zero? ? 1 : -1
end
def to_f
if exponent == 31 # special value in binary16 - all exponent bits are 1
return fraction.zero? ? (sign * Float::INFINITY) : Float::NAN
end
sign * 2**(exponent - 15) * (1.0 + fraction.to_f / 1024)
end
end
What I would like:
Binary16Be.read("\x3C\x00")
=> 1.0
What happens right now:
Binary16Be.read("\x3C\x00")
{:sign_bit=>0, :exponent=>15, :fraction=>0}
(this is not actually my own answer, I received this from the gem's author. Made slight alterations to his answer to fit this Q&A format a little better.)
The procedure is described in the bindata Wiki / Primitive Types
In your case:
Subclass Primitive instead of Record
Rename #to_f to #get
Implement #set
The converted code
class Binary16Be < BinData::Primitive
# naming based on
# https://en.wikipedia.org/wiki/Half-precision_floating-point_format
bit1 :sign_bit
bit5 :exponent
bit10 :fraction
def sign
sign_bit.zero? ? 1 : -1
end
def get
if exponent == 31 # special value in binary16 - all exponent bits are 1
return fraction.zero? ? (sign * Float::INFINITY) : Float::NAN
end
sign * 2**(exponent - 15) * (1.0 + fraction.to_f / 1024)
end
def set(val)
self.sign = (val >= 0.0)
self.fraction = ... # TODO implement
self.exponent = ... # TODO implement
end
end

How to parse ruby BigDecimal inspect?

In the following code:
x = BigDecimal(10)
s = x.inspect # "#<BigDecimal:6fe4790,'0.1E2',9(36)>"
Is there a way to parse s and get the original value ? The reason is that I have some text files with BigDecimal written in them using inspect, and I need to parse these values.
You .to_s to get the value in string. .inspect will print the object
x = BigDecimal(10)
x.to_s
# => "0.1E2"
The documentation for BigDecimal#inspect is incomplete. Consider the following:
require 'bigdecimal`
BigDecimal.new("1.2345").inspect
#=> "#<BigDecimal:7fb06a110298,'0.12345E1',18(18)>"
...
BigDecimal.new("1.234567890").inspect
#=> "#<BigDecimal:7fb06a16ab58,'0.123456789E1',18(27)>"
BigDecimal.new("1.2345678901").inspect
#=> "#<BigDecimal:7fb06a14a6a0,'0.1234567890 1E1',27(27)>"
BigDecimal.new("1.23456789012").inspect
#=> "#<BigDecimal:7fb06a1393a0,'0.1234567890 12E1',27(27)>"
BigDecimal.new("1.234567890123").inspect
#=> "#<BigDecimal:7fb06a123780,'0.1234567890 123E1',27(27)>"
It can be seen from the source code for inspect that, if there are more than 10 significant digits, each 10 characters are separate by a space (for readability, presumably):
BigDecimal.new("123.456789012345678901234567").inspect
#=> "#<BigDecimal:7fb06a0ac8b0,'0.1234567890 1234567890 1234567E3',36(36)>"
I suggest retrieving the string representation of the BigDecimal value as follows:
str = "#<BigDecimal:7fb06a14a6a0,'0.1234567890 1E1',27(27)>"
str.delete(' ').split(?')[1]
#=> "0.12345678901E1"
We are not finished. We must still convert the string we extract to a numerical object. We cannot use BigDecimal#to_f, however, if the value is large in absolute value:
"1.23456789012345678".to_f
#=> 1.2345678901234567
The safest course of action is to return a BigDecimal object, using the method BigDecimal::new, which takes two arguments:
the value to be converted to a BigDecimal object, which can be an Integer, Float, Rational, BigDecimal, or String. If a String, which is what we will supply, "spaces are ignored and unrecognized characters terminate the value" (similar to "123.4cat".to_f #=> 123.4).
the number of significant digits. If omitted or zero, the number of significant digits is determined from the value. I will omit this argument. (For example, BigDecimal.new("0.1234E2").precs #=> [18, 18], where the array contains the current and maximum numbers of significant digits.
Note the second argument is required if the first is a Float or Rational, else it is optional.
We therefore can write:
require 'bigdecimal'
def convert(str)
BigDecimal.new(str.delete(' ').split(?')[1])
end
convert "#<BigDecimal:7facd39d7ee8,'0.1234E4',9(18)>"
#=> #<BigDecimal:7facd39c7de0,'0.1234E4',9(18)>
convert "#<BigDecimal:7facd39b7be8,'0.1234E2',18(18)>"
#=> #<BigDecimal:7facd39ae610,'0.1234E2',18(18)>
convert "#<BigDecimal:7facd3990638,'0.1234E0',9(18)>"
#=> #<BigDecimal:7facd3980aa8,'0.1234E0',9(18)>
convert "#<BigDecimal:7facd3970e28,'0.1234E-2',9(18)>"
#=> #<BigDecimal:7facd39625d0,'0.1234E-2',9(18)>
v = convert "#<BigDecimal:7fb06a123780,'0.1234567890 123E1',27(27)>"
#=> #<BigDecimal:7fb069851d78,'0.1234567890 123E1',27(27)>
An easy way to see if the BigDecimal object can be converted to a float without loss of accuracy is:
def convert_bd_to_float(bd)
f = bd.to_f
(bd==BigDecimal.new(f.to_s)) ? f : nil
end
convert_bd_to_float BigDecimal.new('1234567890123456')
#=> 1.234567890123456e+15
convert_bd_to_float BigDecimal.new('12345678901234567')
#=> nil
"#<BigDecimal:6fe4790,'0.1E2',9(36)>"[/(?<=').+(?=')/]
# => "0.1E2"
I don't know which version of Ruby you are using, so I checked some MRI source code for BigDecimal:
2000 /* Returns debugging information about the value as a string of comma-separated
2001 * values in angle brackets with a leading #:
2002 *
2003 * BigDecimal.new("1234.5678").inspect ->
2004 * "#<BigDecimal:b7ea1130,'0.12345678E4',8(12)>"
2005 *
2006 * The first part is the address, the second is the value as a string, and
2007 * the final part ss(mm) is the current number of significant digits and the
2008 * maximum number of significant digits, respectively.
2009 */
2010 static VALUE
2011 BigDecimal_inspect(VALUE self)
2012 {
2013 ENTER(5);
2014 Real *vp;
2015 volatile VALUE obj;
2016 size_t nc;
2017 char *psz, *tmp;
2018
2019 GUARD_OBJ(vp, GetVpValue(self, 1));
2020 nc = VpNumOfChars(vp, "E");
2021 nc += (nc + 9) / 10;
2022
2023 obj = rb_str_new(0, nc+256);
2024 psz = RSTRING_PTR(obj);
2025 sprintf(psz, "#<BigDecimal:%"PRIxVALUE",'", self);
2026 tmp = psz + strlen(psz);
2027 VpToString(vp, tmp, 10, 0);
2028 tmp += strlen(tmp);
2029 sprintf(tmp, "',%"PRIuSIZE"(%"PRIuSIZE")>", VpPrec(vp)*VpBaseFig(), VpMaxPrec(vp)*VpBaseFig());
2030 rb_str_resize(obj, strlen(psz));
2031 return obj;
2032 }
2033
So, what you want seems to be the second part of the inspect-string, 0.1E2 in your case, equal to 10. The comment above is quite clear, this should be the full numeric value of the object. Simple regex will be enough.
another option:
"#<BigDecimal:95915c4,'0.1E2',9(27)>".split(",")[1].tr! "'", ''
=> "0.1E2"

Encrypting/decrypting 3DES in Ruby

I have a key.bin file which content is something along the lines of:
-12, 110, 93, 14, -48, ...
This is being used by a service to decrypt 3DES content, but I need to encrypt it via Ruby.
I've tried loads of scenarios with how to set the key and what to do with it, but to no avail as of yet:
Tried splitting the key by , and converting each number to hex, concatenating the hex values to make the key
Tried converting the number string to binary
Tried converting the resulting hex to binary
I assume what I need to do is something simple like:
des = OpenSSL::Cipher::Cipher.new('des3')
des.decrypt
des.key = mistery # this step is where i'm having problems at
final = des.update(encrypted) + des.final
Any ideas on what I should do with this key?
Key sample:
-62,-53,124,-110,37,-88,-48,31,-57,93,70,-101,44,69,-88,-57,-123,-99,118,-119,110,55,11,14
Data sample:
NEb2b9sYXgod6mTvaRv+MRsTJvIiTTI9VjnDGcxjxcN5qBH7FXvxYI6Oj16FeKKsoQvjAmdju2SQ
ifJqPJTl97xeEbjdwm+W8XJnWs99ku85EznVBhzQxI1H2jPiZIcrqIVCWdd/OQun7AjK4w2+5yb7
DPN2OiCIEOz2zK6skJrBw3oTEHpXrSEhydOYxqI+c5hC4z3k5nktN6WSVLIo8EAjwenHPMDxboWF
ET8R+QM5EznVBhzQxI1H2jPiZIcrqIVCWdd/OQun7AjK4w2+5yb7DPN2OiCIFqk4LRwEVq16jvKE
vjz6T4/G34kx6CEx/JdZ1LdvxC3xYQIcwS0wVnmtxorFm4q5QQFHzNKQ5chrGZzDkFzAogsZ2l2B
vcvlwgajNGHmxuVU83Ldn1e5rin7QqpjASqeDGFQHkiSbp4x6axVce2OGgfFpZdzCM7y6jLPpOlX
LOC/Bs1vTwMzcBNRB/Fo4nsX9i4It8Spm228XQNUpQe4i9QGe/4AyKIhHoM8DkXwPZ6rWp0W0UMe
keXjg41cED1JwjAAQSP3RYITB78bu+CEZKPOt2tQ2BvSw55mnFcvjIAYVQxCHliQ4PwgceHrnsZz
5aagC0QJ3oOKw9O0dlkVE3IM6KTBMcuZOZF19nCqxMFacQoDxjJY8tOJoN0Fe4Boz2FPyuggfLz9
yhljVJhxqOlTd8eA34Ex8SdC+5NDByAMumjzcPcXL8YVpSN85gytfd+skXhz3npmJ0dmZZOouu0Z
vMmlaCqw96Sy0L1mHLKbjqmZ/W57OBNRB/Fo4nsX9i4It8Spm228XQNUpQe4i9QGe/4AyKIhHoM8
DkXwPZ5tXdq1dRG6IaS51oNzFFlOoP3wTJuOTpj+zQOBMMOi4ENFyyEwYbG/qE+uY8rVwBOUHv9b
Yd9byvOZbnHDuf4oaWRZ+4K3s2NkEblDF9wU6Mb0ZqnLEJsypjrorH1cNIodIDu8nME1nD5bIDF6
XNrWC6pk6AV6eYQvNJw2QDz0RBD15fz/fAXCvbaCLDnhBKpLXrRbQdV+jxx2ipeC2ceMLLRFRPuR
B+ycYht65lWh4jNjoEsBXGFKiT0bSX6Lx/ZQD3twJWbML8ifRhw7SW0jOkUF+dAfXYNaD6nqA6Xq
TkcsDGaJsVq8wwCIWNh6tDRSw7ba4c391147kmnqEgXdKmmnEzUfHtpRw88C0/u0qj809hB4qB0B
lxj/87aDo4VOz9S4jjtk849CxtA/a9+532A4YlXjsPt/f0KZ2drAGEr1VSWzaLh/sMwP5tznmPaK
uozS6C74gMNdhtNMFz0HONcYecS0hg4lrdRyljROgzC33QoBIHbQXJrG0OXE3+81uhJwusEnFaD9
8Eybjk6YeNk3oxL3C5fx/xXgFmhcLLGdxRe/am0jqA1gV6MyQFUKtzdnNOUYpHkYXT9Ea7YYln4Q
D96Z9AI5EznVBhzQxI1H2jPiZIcrqIVCWdd/OQun7AjK4w2+5yb7DPN2OiCIFqk4LRwEVq16jvKE
vjz6T4/G34kx6CEx/JdZ1LdvxC3iEcYTrEH9kKhPrmPK1cATlB7/W2HfW8rzmW5xw7n+KGlkWfuC
t7NjZBG5QxfcFOjG9GapyxCbMqY66Kx9XDSKHSA7vJzBNZw+WyAxelza1guqZOgFenmElSgtUOo7
TEunuphaMIEQgo0udojG6dm2FtRmA4yntNCnCDzGTY72nrFBz3EZmVXGEm6X3Xd5Ito=
Got it working!
Here's how:
Decryption
def read_key(key_file)
File.read(key_file).split(',').map { |x| x.to_i }.pack('c*')
end
des = OpenSSL::Cipher::Cipher.new('des-ede3')
des.decrypt
des.key = read_key('key.bin')
result = des.update(decoded) + des.final
Encryption
def read_key(key_file)
File.read(key_file).split(',').map { |x| x.to_i }.pack('c*')
end
des2 = OpenSSL::Cipher::Cipher.new('des-ede3')
des2.encrypt
des2.key = read_key('key.bin')
result = des2.update(result) + des2.final
puts Base64.encode64(result)

Determining All Possibilities for a Random String?

I was hoping someone with better math capabilities would assist me in figuring out the total possibilities for a string given it's length and character set.
i.e. [a-f0-9]{6}
What are the possibilities for this pattern of random characters?
It is equal to the number of characters in the set raised to 6th power.
In Python (3.x) interpreter:
>>> len("0123456789abcdef")
16
>>> 16**6
16777216
>>>
EDIT 1:
Why 16.7 million? Well, 000000 ... 999999 = 10^6 = 1M, 16/10 = 1.6 and
>>> 1.6**6
16.77721600000000
* EDIT 2:*
To create a list in Python, do: print(['{0:06x}'.format(i) for i in range(16**6)])
However, this is too huge. Here is a simpler, shorter example:
>>> ['{0:06x}'.format(i) for i in range(100)]
['000000', '000001', '000002', '000003', '000004', '000005', '000006', '000007', '000008', '000009', '00000a', '00000b', '00000c', '00000d', '00000e', '00000f', '000010', '000011', '000012', '000013', '000014', '000015', '000016', '000017', '000018', '000019', '00001a', '00001b', '00001c', '00001d', '00001e', '00001f', '000020', '000021', '000022', '000023', '000024', '000025', '000026', '000027', '000028', '000029', '00002a', '00002b', '00002c', '00002d', '00002e', '00002f', '000030', '000031', '000032', '000033', '000034', '000035', '000036', '000037', '000038', '000039', '00003a', '00003b', '00003c', '00003d', '00003e', '00003f', '000040', '000041', '000042', '000043', '000044', '000045', '000046', '000047', '000048', '000049', '00004a', '00004b', '00004c', '00004d', '00004e', '00004f', '000050', '000051', '000052', '000053', '000054', '000055', '000056', '000057', '000058', '000059', '00005a', '00005b', '00005c', '00005d', '00005e', '00005f', '000060', '000061', '000062', '000063']
>>>
EDIT 3:
As a function:
def generateAllHex(numDigits):
assert(numDigits > 0)
ceiling = 16**numDigits
for i in range(ceiling):
formatStr = '{0:0' + str(numDigits) + 'x}'
print(formatStr.format(i))
This will take a while to print at numDigits = 6.
I recommend dumping this to file instead like so:
def generateAllHex(numDigits, fileName):
assert(numDigits > 0)
ceiling = 16**numDigits
with open(fileName, 'w') as fout:
for i in range(ceiling):
formatStr = '{0:0' + str(numDigits) + 'x}'
fout.write(formatStr.format(i))
If you are just looking for the number of possibilities, the answer is (charset.length)^(length). If you need to actually generate a list of the possibilities, just loop through each character, recursively generating the remainder of the string.
e.g.
void generate(char[] charset, int length)
{
generate("",charset,length);
}
void generate(String prefix, char[] charset, int length)
{
for(int i=0;i<charset.length;i++)
{
if(length==1)
System.out.println(prefix + charset[i]);
else
generate(prefix+i,charset,length-1);
}
}
The number of possibilities is the size of your alphabet, to the power of the size of your string (in the general case, of course)
assuming your string size is 4: _ _ _ _ and your alphabet = { 0 , 1 }:
there are 2 possibilities to put 0 or 1 in the first place, second place and so on.
so it all sums up to: alphabet_size^String_size
first: 000000
last: ffffff
This matches hexadecimal numbers.
For any given set of possible values, the number of permutations is the number of possibilities raised to the power of the number of items.
In this case, that would be 16 to the 6th power, or 16777216 possibilities.

Resources