Decrypting Windows Crypto API RSA Ciphertext With OpenSSL - winapi

I have created a C++ program that encrypts some bytes with a RSA 2048 key.
The output of the program is 256 bytes and everything seems to work fine.
However I've been completely unable to figure out how to decrypt this with openssl.
I've been told that I might need to reverse the byte order so it becomes big endian but it doesn't seem to help. (I assume reversing the bytes order means turning 1F 02 C8 into F1 20 8C etc.)
I'm mainly getting the following error from openssl.
openssl rsautl -in output.bin -inkey privatekey -decrypt
rsa routines:RSA_padding_check_PKCS1_type_2:block type is not 02:rsa_pk1.c:190:
Here is a code snippet of my encryption for anyone that interested (I've tried it with final as true also)
BYTE crapBuffer[1200] = {0x33,0x1F,0x22,0x34,0x33,0x1F,0x22,0x34,0x33,0x1F,0x22,0x34,0x33,0x1F,0x22,0x34,0x33,0x1F,0x22,0x34};
DWORD dataLen = 20;
CryptEncrypt(publicKey,0,false,0,(BYTE *)crapBuffer,&dataLen,1200);
Any help would be greatly appreciated,

Related

Find public elliptic curve key using OpenSSL CLI

I can easily generate a new pair
$ openssl ecparam -genkey -name secp256k1 -param_enc explicit | openssl pkey -noout -text
Private-Key: (256 bit)
priv:
61:98:b4:78:a4:59:47:86:ef:2d:aa:4f:2a:21:91:
60:11:18:ff:8f:79:83:c6:f7:d6:1c:b8:9f:1a:85:
1e:47
pub:
04:4f:c4:07:1e:f1:dd:56:39:61:31:2b:75:53:68:
e5:c4:99:53:67:af:da:6b:09:75:2b:a4:d3:3a:40:
32:50:b3:fa:23:05:cc:83:bb:b0:f7:76:30:37:64:
df:47:5f:fa:49:b3:39:9e:3c:8b:34:59:5b:23:fb:
91:7a:1b:73:6b
Field Type: prime-field
Prime:
00:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:
ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:fe:ff:
ff:fc:2f
A: 0
B: 7 (0x7)
Generator (uncompressed):
04:79:be:66:7e:f9:dc:bb:ac:55:a0:62:95:ce:87:
0b:07:02:9b:fc:db:2d:ce:28:d9:59:f2:81:5b:16:
f8:17:98:48:3a:da:77:26:a3:c4:65:5d:a4:fb:fc:
0e:11:08:a8:fd:17:b4:48:a6:85:54:19:9c:47:d0:
8f:fb:10:d4:b8
Order:
00:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:
ff:fe:ba:ae:dc:e6:af:48:a0:3b:bf:d2:5e:8c:d0:
36:41:41
Cofactor: 1 (0x1)
but how do I for a given private ECDSA key
18e14a7b6a307f426a94f8114701e7c8e774e7f9a47e2c2035db29a206321725
find its paired public key?
0250863ad64a87ae8a2fe83c1af1a8403cb53f53e486d8511dad8a04887e5b2352
As opposed to coding, can this be achieved by means of OpenSSL CLI and perhaps other standard Linux utilities?
I tried following man ec which only got me this far:
$ openssl ec -modulus
ec: Unrecognized flag modulus
ec: Use -help for summary.
Then I thought of adding a custom priv to the specified parameters for -genkey to generate the key from. But to even try this one, I'm not sure how to avoid messing directly with DER file format.
$ openssl version
OpenSSL 1.1.1 11 Sep 2018

Generate OpenSSL subject_name hash in ruby

I've investigated 30261296 however I'm still at a loss to find a way to generate the same results in Ruby with the openssl and/or digest gems. The OpenSSL output I'm trying to replicate in ruby is as follows:
$ openssl x509 -noout -subject_hash -in DigiCertSHA2SecureServerCA.pem
85cf5865
In reading many things, I believe this hash is generated from the Subject: portion of the certificate, which is like the distinguished name. In this certificates case something to the effect of:
$ openssl x509 -noout -subject -in DigiCertSHA2SecureServerCA.crt
subject=C = US, O = DigiCert Inc, CN = DigiCert SHA2 Secure Server CA
Attempting to SHA-1 encode that on the command line or in Ruby (which represents this as /C=US,/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA when using the openssl gem) has not yeilded the same has results displayed by OpenSSL.
I'm trying to do this more natively in Ruby to avoid shelling out to openssl if possible since openssl and digest come along with the ruby env. In the end I need this to generate the hash directory tree ... i.e. 85cf5865.0 (hash + '.0').
The CA I'm hasing is DigiCertSHA2SecureServerCA.crt - DER encoded. I converted DER to PEM because openssl command line uses that without the additional -inform der switch. It doesn't appear to matter to Ruby's openssl gem.
This turns out to be pretty straightforward, since Ruby’s OpenSSL bindings includes the OpenSSL::X509::Name#hash method, which is exactly what we want.
require 'openssl'
# Read the certificate.
cert = OpenSSL::X509::Certificate.new(File.binread("DigiCertSHA2SecureServerCA.crt"))
# Get the subject, which is an OpenSSL::X509::Name object.
name = cert.subject
# hash returns an integer, we want the hex string so call to_s(16).
puts name.hash.to_s(16) #=> 85cf5865
The integer will be positive since OpenSSL returns an unsigned int, so we can just use to_s(16) and don’t need to worry about converting negative values.

OpenSSL key length too short in Ruby, not in Bash

I originally experimented with a simple encryption script in Bash and it worked pretty much as expected. However, I'm now trying to do the same thing in Ruby and the Ruby version seems function a little differently.
Bash
Encrypt
echo 'hello' | openssl enc -aes-256-cbc -a
Password: mypass
Result: U2FsdGVkX19rERfOXiKs97FgwIkLy3+ttZzaHkEoQyE=
Decrypt
echo 'U2FsdGVkX19rERfOXiKs97FgwIkLy3+ttZzaHkEoQyE=' | openssl aes-256-cbc -d -a
Password: mypass
Result: hello
Ruby
require "openssl"
require 'base64'
cipher = OpenSSL::Cipher.new('AES-256-CBC').encrypt
cipher.key = 'mypass'
This is what I've attempted in Ruby so far but I receive a OpenSSL::Cipher::CipherError: key length too short error. I would like to mimic Bash as much as possible.
OpenSSL uses a (largely undocumented) password based key derivation function (PBKDF) called EVP_BytesToKey using an 8 byte salt and an iteration count of 1. A magic and salt of 8 bytes each are prefixed to the ciphertext (check the first bytes of the result to see the magic).
Obviously "mypass" cannot be a correct key for AES. AES keys are 16, 24 or 32 byte binary values for the 128, 192 and 256 key sizes. You can however specify a key directly using the -K switch on the command line to make the code compatible with the Ruby Cipher object. In that case you need to specify the key using binary (a file) or hexadecimals for the openssl command line and in Ruby. You would also need to specify an IV.
Alternatively you would have to find an EVP_BytesToKey implementation for Ruby, but note that this is an old OpenSSL specific function with a completely insecure iteration count.

Unable to load Private Key

I am new to SSL/OpenSSL and I'm working on Windows 7. I'm trying to configure HTTPS for my ElasticBeanstalk environment following these instructions.
I'm at Step 2 in "Create a Private Key". After I issue the command to generate the key pair:
openssl genrsa 2048 > privatekey.pem
I get:
Generating RSA private key, 2048 bit long modulus
........................................+++
...............................+++
unable to write 'random state'
e is 65537 (0x10001)
However, it does write a key to my directory. But after the second command:
openssl req -new -key privatekey.pem -out csr.pem
I get:
unable to load Private Key
6312:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:647:Expecting: ANY PRIVATE KEY
I've tried Googling this a bit, but none of the solutions I've found seem to be relevant for me. I checked the generated key and it looks like
-----BEGIN RSA PRIVATE KEY-----
{lots of characters}
-----END RSA PRIVATE KEY-----
What am I doing incorrectly?
unable to load Private Key
6312:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:647:Expecting: ANY PRIVATE KEY
I ran your commands on OS X, and I could not reproduce the results.
I did use the -config option because I have an "OpenSSL server config template" that makes it easy to generate CSRs and self signed certificates:
$ mkdir test
$ cd test
$ openssl req -new -key privatekey.pem -out csr.pem -config example-com.conf
The configuration file is named example-com.conf, and you can find it at How do I edit a self signed certificate created using openssl xampp?. Edit it to suit your taste (in particular, the DNS names).
If interested, here's the OpenSSL man pages on the req sub-command.
I checked the generated key and it looks like
-----BEGIN RSA PRIVATE KEY----- {lots of characters}
-----END RSA PRIVATE KEY-----
You can validate the key you just created with:
$ openssl rsa -in privatekey.pem -inform PEM -text -noout
Private-Key: (2048 bit)
modulus:
00:b0:91:ce:57:28:0f:5c:3a:c3:29:d7:23:6a:71:
ca:64:49:fc:24:ea:69:a3:09:d6:49:94:17:b9:09:
65:fa:5a:10:47:a4:9b:b8:cd:6d:32:74:19:8d:5c:
79:92:f0:a6:43:9c:75:a3:7b:ef:c4:c3:d9:c2:db:
b9:bd:ec:14:a8:b1:52:73:8f:56:c8:5c:16:08:56:
ff:c2:2b:35:3c:0a:0f:34:d0:91:c1:54:7e:72:e8:
97:bf:ea:46:69:5f:e4:21:8d:7a:f5:a5:6b:6a:e8:
00:56:bc:02:f6:b4:ae:6e:89:a6:50:aa:5b:2f:d8:
7d:99:04:61:51:76:b3:5e:9e:30:52:99:54:26:e2:
3a:54:ec:78:34:e6:9a:b7:c2:58:5c:51:3d:39:52:
d4:6e:0c:6e:a1:a0:a5:f1:4d:5a:f5:0b:1a:6e:dc:
f3:bb:0d:d0:53:51:b0:1a:04:ee:86:35:d5:f3:8b:
0d:bc:19:61:6c:0c:b2:7b:a9:7c:47:97:01:bb:a2:
6a:74:d9:19:e9:df:60:07:d4:95:4c:83:f8:3b:84:
c2:b8:3d:b9:a7:34:0a:9b:a3:c6:70:cc:ef:de:f4:
64:88:f1:56:d3:2a:fd:5a:82:88:96:66:93:6c:a0:
b8:ec:e4:4c:e8:76:5c:9c:fc:c4:60:72:b6:9a:3f:
98:a3
publicExponent: 65537 (0x10001)
privateExponent:
00:87:ab:f1:65:ac:e5:68:93:ca:64:3a:e7:fe:a1:
62:c7:7e:c5:dc:c3:b5:d9:cd:f4:36:e3:30:fb:40:
0a:78:bc:7d:67:df:46:bc:50:34:88:a1:07:05:44:
ba:31:ba:f1:b6:5f:e1:50:76:29:bd:02:54:2f:d2:
cf:bc:ec:4a:cf:78:39:07:8c:6b:3d:56:ec:a3:09:
de:49:9f:13:af:87:77:39:b8:cd:56:45:0b:48:56:
0a:4c:2f:c2:5c:b3:8e:c2:6d:48:be:b9:95:79:36:
bd:13:e8:31:4a:c9:78:82:7d:08:2b:51:4a:f1:cf:
a2:6a:52:20:49:0d:31:34:10:88:02:d7:a7:07:70:
32:b5:f5:8c:cc:d4:b2:8d:b9:aa:bb:33:82:1a:74:
bd:4d:4f:e9:e0:cc:f2:27:fb:98:34:2c:77:56:6f:
88:3a:66:32:5d:7d:57:c6:5b:63:39:fa:32:04:9d:
e3:cc:a5:b6:44:91:fd:7d:d1:b6:2d:16:47:59:81:
3d:cf:d9:a7:58:2a:d6:61:5d:c6:69:3b:7a:70:50:
4f:80:f4:d9:fb:c8:7d:5e:44:9e:ac:c8:e6:aa:49:
c3:d6:df:6b:03:68:25:a3:2b:89:8f:9a:35:3a:58:
7d:71:b4:08:d9:04:7b:b9:96:17:f3:a5:19:c5:07:
4e:c1
prime1:
00:d7:d0:d8:8c:b5:86:ed:0e:06:70:c9:54:00:25:
d7:8c:e4:65:51:1b:c5:ba:33:c2:02:1a:dc:80:a6:
ae:8e:1e:e8:c0:b7:04:11:5a:e3:98:52:8f:4a:7a:
43:b8:e8:1b:c8:d6:d3:b2:dc:70:59:a5:ca:83:bb:
35:f1:6c:f5:cb:d0:f4:04:5e:aa:7c:d0:ec:d7:4a:
d5:1c:7c:e2:67:e4:e8:17:95:9b:4e:2b:a0:26:74:
61:d0:a0:15:27:18:e5:84:b5:54:ef:be:82:35:7e:
78:e0:49:6b:4e:ae:93:53:a0:81:a3:8e:de:d3:e5:
dc:c5:ba:03:36:14:47:97:03
prime2:
00:d1:72:3b:f5:34:b1:11:78:b2:79:f4:3e:d7:be:
bf:cc:b3:09:ea:24:a4:cc:7f:64:73:96:d2:48:9e:
55:bc:79:23:c2:d9:80:81:7d:a4:a5:4b:43:33:8e:
62:04:ec:8d:22:d7:43:5e:41:b6:4d:e9:b0:cc:70:
63:17:70:93:88:81:f5:84:a6:3f:2b:98:33:a3:69:
53:11:c7:95:8c:30:ea:e8:58:c7:77:10:b4:a8:f5:
bf:5e:cf:e1:99:bb:b3:4e:57:d2:4c:f7:73:de:8a:
98:8e:7c:26:37:6c:e4:77:c6:d2:ed:5d:53:a7:15:
c3:9c:67:61:d3:24:9a:f5:e1
exponent1:
00:83:34:59:e2:b9:9d:8c:d2:e1:01:82:b4:89:de:
77:bc:15:42:af:5b:c6:0a:dc:da:8e:f3:0b:a9:3f:
2c:92:04:a2:96:3e:ed:bf:2b:55:80:ce:78:84:db:
ed:fe:25:46:77:04:7b:f1:9a:68:c7:67:ae:c6:05:
73:d7:11:da:21:0e:28:bb:db:5d:a4:c2:53:aa:d3:
b8:da:37:e6:61:29:5e:1c:b0:7c:99:ba:96:03:aa:
ef:a8:a9:1a:13:09:e4:c7:98:82:49:ba:b5:68:96:
3a:20:89:22:2e:d4:9d:86:d2:e6:dd:ab:c7:36:65:
e1:a1:67:e3:f9:e5:bc:5c:47
exponent2:
00:81:6d:b9:55:8f:09:39:05:c0:2d:12:dd:5e:cf:
56:91:35:b6:93:c5:af:3d:5c:20:04:3a:18:9a:9d:
95:d7:d1:78:62:e9:ab:ba:d9:9c:cc:34:95:43:9f:
e2:3c:ae:bd:8c:e1:3f:95:58:c0:42:a7:7e:04:e8:
12:a4:22:82:59:22:0e:49:b9:be:61:bf:3d:71:e7:
1d:59:68:5f:a6:f1:77:c8:bb:4c:0f:ec:f7:e7:4d:
6d:c4:36:6c:70:67:08:a8:0a:27:40:3e:ce:90:a0:
4f:24:05:de:4b:f3:f3:bf:7c:d3:4d:b1:95:87:34:
30:dc:4f:1a:a9:b2:fe:3b:a1
coefficient:
6d:51:b3:6e:87:8d:aa:f0:55:c4:22:21:62:a9:ea:
24:b3:b7:91:40:f5:78:5d:f1:40:45:7e:0d:a2:a3:
54:46:ba:42:33:b6:cd:57:a1:85:bc:3d:ba:1c:eb:
87:33:a9:e9:63:1e:7c:2c:89:98:b9:0f:4b:e8:c4:
79:bd:00:6a:f5:3e:ea:63:f1:9e:aa:47:35:5a:22:
fc:4e:e3:61:7e:eb:dc:a6:c0:2c:d5:fd:22:9f:01:
59:32:15:db:41:99:b7:a8:c1:eb:1e:42:c7:1b:c7:
c8:56:86:a8:34:fe:1c:48:b6:6e:f1:c1:5c:96:bf:
9d:fa:e5:4c:d0:2a:d9:09
unable to write 'random state'
This is a well known problem. OpenSSL uses a default configuration file. You can locate the configuration file with correct location of openssl.cnf file.
The default configuration file includes these lines:
$ cat /usr/local/ssl/macosx-x64/openssl.cnf
...
HOME = .
RANDFILE = $ENV::HOME/.rnd
...
To save the random file, you should point HOME and RANDFILE to a valid location. On Windows, you type set HOME=... and set RANDFILE=... in the command prompt. Or better, change it in the OpenSSL configuration file you use.
Also see How to fix “unable to write 'random state' ” in openssl and How do I make OpenSSL write the RANDFILE on Windows Vista?.
I'm trying to configure HTTPS for my ElasticBeanstalk environment following these instructions.
The instructions are wrong in the image below. Do not place a DNS name in the Common Name (CN).
Placing a DNS name in the Common Name is deprecated by both the IETF (the folks who publish RFCs) and the CA/B Forums (the cartel where browsers and CAs collude). You should pay articular attention to what the CA/B recommends because Browsers and CAs come up with those rules, and the browsers follow them (and they don't follow the RFCs). For reference, see RFC 5280, RFC 6125 and the CA/B Baseline Requirements.
Instead, place DNS names in the Subject Alternate Name (SAN). Both the IETF and CA/B specifies it.
The custom OpenSSL configuration file handles this for you. You just have to change the DNS names listed under the section [ alternate_names ]. For example, here's a set of names set up for the domain example.com. Notice there is no DNS name in the CN:
[ subject ]
...
commonName = Common Name (e.g. server FQDN or YOUR name)
commonName_default = Example Company
[ alternate_names ]
DNS.1 = example.com
DNS.2 = www.example.com
DNS.3 = mail.example.com
DNS.4 = ftp.example.com
Can you check if you have appropriate permissions when you run both the commands? Maybe try doing the same using a user with Admin Rights.
Also make sure the created file privatekey.pem has appropriate permissions before executing the command below (Use chmod if necessary)
openssl req -new -key privatekey.pem -out csr.pem
Submitting this as answer as I don't have enough reputation to comment.
I believe the root of the problem is the error
unable to write 'random state'
e is 65537 (0x10001)
Searching StackOverflow found these results. I would stress that you run the openssl program as sudo or directly as root to avoid any possible permissions issues.
The fix in Windows:
https://stackoverflow.com/a/12522479/3765769
In Linux:
https://stackoverflow.com/a/94458/3765769

openssl AES encryption adds a blocksize worth of bytes to output

I'm trying to encrypt with openssl on the console to match output generated by another implementation of AES. All details are known. I'm using AES in 128-bit CBC mode. Weirdly enough, irrespective of the file size, the output will be 16 bytes larger. I think openssl is appending some kind of padding.
In the direction to the other implementation it's not that much of a problem as I can drop the last 16 bytes, but the other way around is as I can't invent the bytes that openssl will probably check for validity.
How do I tell openssl not to do that?
Commandline:
openssl enc -aes-128-cbc -K <pre-shared key in hex> -in rawfile.bin -out encfile.enc -iv <pre-shared IV in hex>
openssl enc has a -nopad option. I've not used it, but it sounds relevant.
-nopad
disable standard block padding

Resources