Ruby SHA1 RSA signing different from command line OpenSSL? - ruby

I'm trying to implement RSA SHA1 signature verification.
To my surprise, command line OpenSSL tool doesn't generate the same key as the Ruby OpenSSL.
If I run those commands :
MacBook-Pro-de-Geoffrey:ssl_tests Escaflowne$ cat data.txt
000
MacBook-Pro-de-Geoffrey:ssl_tests Escaflowne$ openssl dgst -sha1 -binary -sign prvkey.pem -out sig.bin data.txt
MacBook-Pro-de-Geoffrey:ssl_tests Escaflowne$ openssl base64 -in sig.bin -out sig64.txt
MacBook-Pro-de-Geoffrey:ssl_tests Escaflowne$ cat sig64.txt
AJEh2kA7O3j624Kdl7UCGN1HiEk/v2LQudB+cjxw1CfmRTjcSPBjUE/EAwy8NEut
K4zYgfRwwTs7NY3AwYiUEtAe5yohUM0Qv17qSDW+G4IWjwe9PKE7Sl00umiMdszA
q/1hqeQlHKgjme7YO7H6i1UcAXmriOOjn+ySRaovsHw=
So final base64 result in command line is :
AJEh2kA7O3j624Kdl7UCGN1HiEk/v2LQudB+cjxw1CfmRTjcSPBjUE/EAwy8NEut
K4zYgfRwwTs7NY3AwYiUEtAe5yohUM0Qv17qSDW+G4IWjwe9PKE7Sl00umiMdszA
q/1hqeQlHKgjme7YO7H6i1UcAXmriOOjn+ySRaovsHw=
Now, if I try signing it through my ruby script :
def sign_message(message)
privkey = OpenSSL::PKey::RSA.new(File.read(Rails.root.join('lib', 'payment', 'prvkey.pem')))
digest = OpenSSL::Digest.new('sha1')
expected_sign = privkey.sign(digest, message)
base_64_expected_sign = [expected_sign].pack('m')
puts "Expected Signature"
puts expected_sign
puts "Base 64 Expected Signature"
puts base_64_expected_sign
return base_64_expected_sign
end
And calling the function like this :
def test_sign
message = "000"
message_signature = sign_message(message)
puts "Message Signature : #{message_signature}"
puts "Valid : #{verify_signature(message_signature, message)}"
end
I get the output :
Expected Signature
??|??n?^~?T_1Y#??BR??u???k x?*????S?L?:.7
t??tc?)崪? ?}DMp?p2??4?D-f??jT;!e
?k??5??
Base 64 Expected Signature
QjhL1zQoUdGFLVCMg06/CKeE/HdhRTOhJ/p09wkWeK0qD/afsxfcU7tMtDou
Nw3rwXw/5W68XhZ+BK1UXwIxWUDbFYlCUpu6HnWTmI5rC3QP+f50Y8kp5bSq
gQkekH1ETXDmcDKvExeSNKVELWYe3uwTalQ7IWUMyWvnF541rvo=
Message Signature : QjhL1zQoUdGFLVCMg06/CKeE/HdhRTOhJ/p09wkWeK0qD/afsxfcU7tMtDou
Nw3rwXw/5W68XhZ+BK1UXwIxWUDbFYlCUpu6HnWTmI5rC3QP+f50Y8kp5bSq
gQkekH1ETXDmcDKvExeSNKVELWYe3uwTalQ7IWUMyWvnF541rvo=
So final ruby OpenSSL signature is :
QjhL1zQoUdGFLVCMg06/CKeE/HdhRTOhJ/p09wkWeK0qD/afsxfcU7tMtDou
Nw3rwXw/5W68XhZ+BK1UXwIxWUDbFYlCUpu6HnWTmI5rC3QP+f50Y8kp5bSq
gQkekH1ETXDmcDKvExeSNKVELWYe3uwTalQ7IWUMyWvnF541rvo=
Versus command line :
AJEh2kA7O3j624Kdl7UCGN1HiEk/v2LQudB+cjxw1CfmRTjcSPBjUE/EAwy8NEut
K4zYgfRwwTs7NY3AwYiUEtAe5yohUM0Qv17qSDW+G4IWjwe9PKE7Sl00umiMdszA
q/1hqeQlHKgjme7YO7H6i1UcAXmriOOjn+ySRaovsHw=
I've been struggling with this for some time now and I don't understand what could be making a difference!
UPDATE :
Well, apparently results match if I replace my message variable with File.read(Rails.root.join('lib', 'payment', 'data.txt'))
So basically, using a string with the same value as what's in the text file doesn't give the same result.
This means it's encoding related right ?
UPDATE 2 :
So the file says its encoded in us-ascii if I run file -I data.txt
However, if I do message.encoding.name it says its loaded as UTF-8
Also, message.encode('ascii') does not alter the result of the generated signature, it still corresponds with the command line openssl.
As soon as I switch to a string "000".encode('utf-8') or "000".encode('ascii'), the signatures don't match anymore.
So encoding doesn't seem to play a role at all.
How come there's a difference between the exact same content whether it comes from reading a file or written as a string ?

The file data.txt has a trailing newline that you are not taking into account in your code. Using
message = "000\n"
should work.
You could also do
message = File.binread("data.txt")
to make sure you get the exact data as the command line.

Related

How to encode protocol buffer string to binary using protoc

I been trying to encode strings using protoc cli utility.
Noticed that output still contains plain text.
What am i doing wrong?
osboxes#osboxes:~/proto/bin$ cat ./teststring.proto
syntax = "proto2";
message Test2 {
optional string b = 2;
}
echo b:\"my_testing_string\"|./protoc --encode Test2 teststring.proto>result.out
result.out contains:
^R^Qmy_testing_string
protoc versions libprotoc 3.6.0 and libprotoc 2.5.0
Just to formalize in an answer:
The command as written should be fine; the output is protobuf binary - it just resembles text because protobuf uses utf-8 to encode strings, and your content is dominated by a string. However, despite this: the file isn't actually text, and you should usually use a hex viewer or similar if you need to inspect it.
If you want to understand the internals of a file, https://protogen.marcgravell.com/decode is a good resource - it rips an input file or hex string following the protocol rules, and tells you what each byte means (field headers, length prefixes, payloads, etc).
I'm guessing your file is actually:
(hex) 10 11 6D 79 5F etc
i.e. 0x10 = "field 2, length prefixed", 0x11 = 17 (the payload length, encoded as varint), then "my_testing_string" encoded as 17 bytes of UTF8.
protoc --proto_path=${protobuf_path} --encode=${protobuf_message} ${protobuf_file} < ${source_file} > ${output_file}
and in this case:
protoc --proto_path=~/proto/bin --encode="Test2" ~/proto/bin/teststring.proto < ${source.txt} > ./output.bin
or:
cat b:\"my_testing_string\" | protoc --proto_path=~/proto/bin --encode="Test2" ~/proto/bin/teststring.proto > ./output.bin

Parsing a certificate string in go

I'm using ssldump to extract the certificate in a communication. When i parse the result I obtain a string in go defined as:
var string certStr
certStr = "30 82 06 9f...."
How can I parse it to a X509 certificate?
UPDATED
I have tried to parse it directly:
certSlc := []byte(certStr)
cert,err := x509.ParseCertificates(certSlc)
But the result was:
Error:asn1: structure error: tags don't match (16 vs {class:0 tag:19 length:48 isCompound:true}) {optional:false explicit:false application:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false}
Should I do another kind of conversion? maybe is the string incomplete or has got wrong type of cert?
I found the error. The problem was in the source.
As I was explaining, my cert string was "30 82 06 09...". This source must be decoded with:
hex.DecodeString(certStr)
The problem is that hex decoding doesn't work with this format. The error I obtained was:
encoding/hex: invalid byte: U+0020 ' '
So, removing whitespaces and carriage returns in the original string is the solution to make it work.
After decoding in a byte slice the X509 certificate can be created with no problem.

Ruby OpenSSL nested asn1 error

I have tried the advice on several of the questions posted here, but to no avail. I have the following files: (NOTE, I generated these on the fly and they are throwaway keys)
cert file:
-----BEGIN CERTIFICATE-----
MIIE+jCCA+KgAwIBAgIJAMLMeL/HH75vMA0GCSqGSIb3DQEBBQUAMIGuMQswCQYD
VQQGEwJVUzENMAsGA1UECBMEVXRhaDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkx
HjAcBgNVBAoTFU5pY2hvbGFzIFRlcnJ5IGRvdGNvbTEUMBIGA1UECxMLZGV2ZWxv
cG1lbnQxFjAUBgNVBAMTDW5pY2hvbGFzdGVycnkxKTAnBgkqhkiG9w0BCQEWGm5p
Y2hvbGFzQG5pY2hvbGFzdGVycnkuY29tMB4XDTE0MTIyNTA3MTkxNFoXDTE1MTIy
NTA3MTkxNFowga4xCzAJBgNVBAYTAlVTMQ0wCwYDVQQIEwRVdGFoMRcwFQYDVQQH
Ew5TYWx0IExha2UgQ2l0eTEeMBwGA1UEChMVTmljaG9sYXMgVGVycnkgZG90Y29t
MRQwEgYDVQQLEwtkZXZlbG9wbWVudDEWMBQGA1UEAxMNbmljaG9sYXN0ZXJyeTEp
MCcGCSqGSIb3DQEJARYabmljaG9sYXNAbmljaG9sYXN0ZXJyeS5jb20wggEiMA0G
CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDRA4sMOx8cn59h8xvE1EcwudGYgGID
WixEEz+8UczLzoIovbZI//iExrwfM7edYRU/HyIufx+dvHj5bRABt5pOwgEiolDA
XtIttxpsCX23gLg/40TQY2JXRv10JgxEQ3680ypghfAEXisOiIQGrT0W+k6WpIRo
/+z4zh/UPqUmvV3ZnzG97jsFiQ0Nu8IKlByLWzoef+2MqG0GO95do3gCcBhAF3tm
v+buNVi8DgJnYTnKKD5S1+KYRSkNNioRs3zE+/W0vCA0/kLiFjSjMq3bSuGJbHmt
N/8+lodfZYh1HB6DRZoyfTynNDRHyKpPGqTvfp8U/E91kkpxusVSQFbrAgMBAAGj
ggEXMIIBEzAdBgNVHQ4EFgQUPAm5VdPIPbdedLtCEmpivYi8xfkwgeMGA1UdIwSB
2zCB2IAUPAm5VdPIPbdedLtCEmpivYi8xfmhgbSkgbEwga4xCzAJBgNVBAYTAlVT
MQ0wCwYDVQQIEwRVdGFoMRcwFQYDVQQHEw5TYWx0IExha2UgQ2l0eTEeMBwGA1UE
ChMVTmljaG9sYXMgVGVycnkgZG90Y29tMRQwEgYDVQQLEwtkZXZlbG9wbWVudDEW
MBQGA1UEAxMNbmljaG9sYXN0ZXJyeTEpMCcGCSqGSIb3DQEJARYabmljaG9sYXNA
bmljaG9sYXN0ZXJyeS5jb22CCQDCzHi/xx++bzAMBgNVHRMEBTADAQH/MA0GCSqG
SIb3DQEBBQUAA4IBAQBIGsrZaWDwW2UWzTT2tbMEPOFVVZFZiWNazJLXFrXO1deW
1Ggaa2+h3tE+dMincQp7KPDeEOZ4VeXjrwZ1UfmJm5qOWhwJ2HnSBHay22tKO24Y
1qDF/ygMqW3yyr10MWZJaxQeUn+cgpclesS8IiTUVF61pYpjI8DrETa8Hwd8QJkC
YaB3qXOVWdbvJX3DTkYPDEqLkC0qzqibe8fWd0JFF5lDMAeCS5EgW+lu6M++dQve
u5L6Y1Xh8SvRB8SnZdy30OKNf0RrWPgNq9donv8nWgEJEYl9A/XRhWvl6ZWpQLEK
6GbCprqUIFgnWBhp0fteIMcgXZUe/oGBQ3/Yovd+
-----END CERTIFICATE-----
private key:
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEA0QOLDDsfHJ+fYfMbxNRHMLnRmIBiA1osRBM/vFHMy86CKL22
SP/4hMa8HzO3nWEVPx8iLn8fnbx4+W0QAbeaTsIBIqJQwF7SLbcabAl9t4C4P+NE
0GNiV0b9dCYMREN+vNMqYIXwBF4rDoiEBq09FvpOlqSEaP/s+M4f1D6lJr1d2Z8x
ve47BYkNDbvCCpQci1s6Hn/tjKhtBjveXaN4AnAYQBd7Zr/m7jVYvA4CZ2E5yig+
UtfimEUpDTYqEbN8xPv1tLwgNP5C4hY0ozKt20rhiWx5rTf/PpaHX2WIdRweg0Wa
Mn08pzQ0R8iqTxqk736fFPxPdZJKcbrFUkBW6wIDAQABAoIBAEFpWHTFc+EjW1/u
EzywKm9nV97gHsxpxfywAXxQJUWLJVTWultyMDZtc6ZYKxiHr3yHo8zlX+GfgESf
Cyleal5HfM93+MmbYy+HZC93cO6izAbCe2C0GayyvNNCrQgYD2vMsjBu+kSDq/nq
Y1crlDjCsSGX7xBlN6ZN68wiptDOVa1aBL+H3kzHtarURVdKY1MXrscR760/o8Nh
qA/9GA1PoHQ7ApNG8oLVPS9SU4JU3PfURCQTk8otYZ4GrScpSeYp9ISi/UODlF1M
IY+B9Bfrf2Eftv8zP+eF1UbTeBmMq0boy+afZsugMqjN/FpUpxUALlRX96az+VY9
fDSuUPECgYEA9Vv8E4HBbnzPrzA6jyYNcSnkfzTcNxuYaIqTrxWSLi018u1kRa9Z
zLyKFNZRg3+RTWKAqDLKkC/oZZARlPcpx8uzV1T+zrffrnf2aK1LDHUHEmsMj7PY
2DIhghO90BPGuWPpbHjmpr51hHH6+n/LcLN0EHQ2T9wh5HZY75qup9cCgYEA2hQL
sE0+Bp2rFP2CMfl5SAhhtmjrTGg76mItMmGj/3ycZ6CxRw/5nSepau7nbCZXSK+i
Pp/EutuUNcD9ZmyP7z7In2PdWP8X8OSjd9abUn3cRPz8IjSOkbuUMKYWnweUl5Qw
IgygFDYYGjEclOuUlbIzzhp9zeOJAwY0vsxulw0CgYAeaEnzOO95++nZMkbvmq2r
yp9QzIJGKhtXSWVIG4pEQsIe2yDEKhkc8HjEYFM10sd1KbH8Jl9IQ0ev3ozvQzpg
UnRlbFkv0UXdX2ygSGm2n4JC3BVwcb97+6p/bmbltK26KBGzqcAcBhqWUXHjPZc+
3l6R83UPrJ5eq/QVrbvbfQKBgBwvNwuEEB4exnuh/++hhHEw1CAVE0P+vK+gHKNE
R0O/wR8Lf53ljKco9xZg5fvuQJ2eRWO+llhoBJGl7ZoNMdUD0j33VCYqYde6VP4p
+E0DAzLPV268SCjBi3d2H7pR6nxkpIviAdZa32aTLlR41e066MMcXWH8pDqF1M9L
8IcJAoGBAMgoNz/y5LOclcfTLsj/1eYYDDCyB0a2rDsykcYSxeUOibRSicIQnE09
VW3NHZS56gsz4LFCK0Jxq9w76u/ZnhU5FCHs0+BwzyDYnyUnZQre38hcbEcMbEoe
Scmh3qRTfoS74qJzxx/rfhqRnLQal/FR4qf8V559gmB8idZmgMwK
-----END RSA PRIVATE KEY-----
The md5s match for these and when I run the following command:
openssl pkcs12 -export -out azure2.pfx -inkey azure2.key -in azure2.crt
I get a binary encoded pkcs12 file back.
However, when i try to do an http post, i get the following error:
Uncaught exception: nested asn1 error
Any ideas?
EDIT:
I tried the following:
p12 = OpenSSL::PKCS12.new(File.read("azure2.pfx"))
p p12.certificate
p p12.ca_certs
and i get valid output.... So what gives?
So, once again, I answered my own question. I was trying to read a DER-form pkcs12. I had to convert it to PEM format.
For others who might run into this. I had this error and fixed it by removing all carriage returns from my certificate string. (eg. replace "\r\n" and "\n\r" with just "\n" or just remove all "\r"

Using ruby SAX parsers for GB2312 encoded xml

Good day,
I have a lot of big xml files that i need to parse, but problems is they have 'gb2312' encoding. I would normaly use SAX parser for this.
So here is in example of xml:
<?xml version="1.0" encoding="gb2312"?>
<Root>
<ValueList Count="112290" FieldCount="11">
<Item1 Value1="23743" Value2="Дипломатия � Пустой кувшин" Value3="1" Value4="" Value5="6" Value6="0" Value7="0" Value8="0" Value9="0" Value10="0" Value11="0"/>
<Item2 Value1="6611" Value2="ДЛ � 018 омела � золотой кинжал" Value3="1" Value4="" Value5="6" Value6="0" Value7="0" Value8="0" Value9="0" Value10="0" Value11="0"/>
<Item3 Value1="6608" Value2="Наука (ДЛ)�круг фей 021�тяпка" Value3="1" Value4="" Value5="6" Value6="0" Value7="0" Value8="0" Value9="0" Value10="0" Value11="0"/>
<Item4 Value1="6612" Value2="Знаки ДЛ � 003руны � разрушение" Value3="1" Value4="" Value5="6" Value6="0" Value7="0" Value8="0" Value9="0" Value10="0" Value11="0"/>
....
</Root>
I'm trying to use Nokogiri SAX (also tried libxml-ruby with same result) parser:
require 'nokogiri'
class SchemaParser < Nokogiri::XML::SAX::Document
def initialize
#cnt = 0
end
def start_element name, attrs =[]
if name == "Item1"
#cnt+= 1
puts #cnt
end
end
end
parser = Nokogiri::XML::SAX::Parser.new(SchemaParser.new)
parser.parse_io(File.open('2_4_EQUIPMENT_ESSENCE.xml'), 'gb2312')
But this gives error "`check_encoding': 'GB2312' is not a valid encoding (ArgumentError)". If I remove encoding declaration and let Nokogiri detect encoding himself, I will receive this error:
encoding error : input conversion failed due to input error, bytes 0xA8 0x43 0x20 0xA7
encoding error : input conversion failed due to input error, bytes 0xA8 0x43 0x20 0xA7
I/O error : encoder error
I also tried to open File with proper encoding, but that didn't help SAX parser:
[3] pry(main)> f = File.open('2_4_EQUIPMENT_ESSENCE.xml', "r:gb2312")
=> #<File:2_4_EQUIPMENT_ESSENCE.xml>
[4] pry(main)> f.external_encoding.name
=> "GB2312"
Did anyone use 'gb2312' encoding with SAX parsers in ruby? Any recommendations how to proceed?
It seems the issue is that Libxml2 does not support the GB2312 encoding (see here for a list of supported encodings).
I'm not sure if you have tried this, but I think you can work around this by removing the encoding declaration from the XML files (so Libxml2 does not try to transcode the data) and set the external encoding of the File object to GB2312, because then Ruby will transcode the file to UTF-8 as it is read, and from then on everything will remain as UTF-8.
So, here is my workaround.
Problems:
Some of characters presented in xml are not 'gb2312' encoding, I have found that 'GB18030' would be a better choice with full Chinese characters.
I converted all xml's to utf8, so i can use SAX parser.
I ended up with this rake task:
desc "convert chinese xml files to utf-8"
task :convert do
rm_rf 'data/utf8'
mkdir 'data/utf8'
Dir.foreach('data') {|f|
if f.end_with?('.xml')
puts "converted:: data/utf8/#{f}" if system("iconv -f GB18030 -t UTF-8 data/#{f} > data/utf8/#{f}")
end
}
#replace encodings for xml files
system("bundle exec ruby -pi -e \"gsub(/gb2312/, 'UTF-8')\" data/utf8/*.xml")
end

Load PKCS#8 binary key into Ruby

I'm trying to load a particular private key encoded in binary DER format (PKCS#8) into Ruby.
However, OpenSSL::PKey won't recognize it. I can make it work by doing some console work and transforming it into a PEM like so:
openssl pkcs8 -inform DER -in file.key -passin pass:xxxxxxxx >private_key.pem
After this, the key can correctly be read.
However, since I would like for the whole process to be done in memory instead of writing and reading files.
So my question is: Is it possible to load private keys from the binary encoded DER format into Ruby/OpenSSL?
Thank you for your time,
Fernando
Yes, you can indirectly load PKCS#8 DER-encoded private keys using Ruby OpenSSL.
OpenSSL::PKey::RSA.new will only handle PEM-formatted PKCS#8, but it is easy to read the binary DER and convert it to a PEM-formatted string and then load from the string.
For example, with these DER-encoded private keys:
$ openssl genrsa | openssl pkcs8 -topk8 -outform DER \
-nocrypt -out pkcs8.key
$ openssl genrsa | openssl pkcs8 -topk8 -outform DER \
-v2 des3 -passout pass:secret -out pkcs8_des3.key
You can do something like this:
require 'openssl'
require 'base64'
def box(tag, lines)
lines.unshift "-----BEGIN #{tag}-----"
lines.push "-----END #{tag}-----"
lines.join("\n")
end
def der_to_pem(tag, der)
box tag, Base64.strict_encode64(der).scan(/.{1,64}/)
end
pem = der_to_pem('PRIVATE KEY', File.read('pkcs8.key'))
key = OpenSSL::PKey::RSA.new(pem)
pem2 = der_to_pem('ENCRYPTED PRIVATE KEY', File.read('pkcs8_des3.key'))
key2 = OpenSSL::PKey::RSA.new(pem2, 'secret')
Read the DER bytes, Base64 them and put the PEM tags on top and bottom, and then load the key.
Certificate is capable of handling DER-encoded certificates and certificates encoded in OpenSSL's PEM format.
You could find documentation about OpenSSL implementation for Ruby here :

Resources