How to check two sided secured https connection using OpenSSL / Curl - windows

Question

The question seems to be asked before:
OpenSSL Verify return code: 20 (unable to get local issuer certificate). However the difference is that it is about a local issuer certificate. Besides, the answers are not for a Windows computer.
Problem description
On a windows computer, I've got a program that tries to contact a secure server. The security is with certificates on both sides.
Problem: I can't contact it, so I tried to find out if the certificates are correctly installed
Searches, a.o. here on stack overflow, indicated that a good method to find problems would be to use OpenSsl for this, even though I'm running a windows computer.
As an example to check if all certificates for a connection are correctly installed, I was advised to check the connection with google.com:
openssl.exe s_client -connect google.com:443
(My browsers have no problems connecting to this server)
The first lines of the response are
CONNECTED(00000184)
depth=1 C = US, O = Google Trust Services, CN = Google Internet Authority G3
verify error:num=20:unable to get local issuer certificate
---
Certificate chain
0 s:/C=US/ST=California/L=Mountain View/O=Google LLC/CN=*.google.com
i:/C=US/O=Google Trust Services/CN=Google Internet Authority G3
1 s:/C=US/O=Google Trust Services/CN=Google Internet Authority G3
i:/OU=GlobalSign Root CA - R2/O=GlobalSign/CN=GlobalSign
---
Server certificate
-----BEGIN CERTIFICATE-----
...
And further down the same error twice:
Verification error: unable to get local issuer certificate
Verify return code: 20 (unable to get local issuer certificate)
Alas the OpenSSL documentation about s_client isn't very informative about these errors.
So what does it mean? Am I missing some certificates to communicate with google.com, or am I using the incorrect program for this?
Of course google.com is just an example. I chose this, so I could check if reported problems are because of certificate problems, or because of the command I use.
For my actual server that I try to contact, I have the proper certificates (up to the root) as .CER files. The root certificate is in the winstore.

Answer 1

Patrick Mevzek pointed me towards the proper answer (Thanks Patrick!). Because some investigation was needed, I decided to write it down as a complete answer.
I'm working in Windows Server 2012. Newer versions will probably work similarly. To test the certificates and the communication I use:
openssl for windows. Find a compiled version here
curl for window here
Files:
So I am a Client of a Server. There is a two-way secure certification: via very secure methods we have the following files:
A Root certificate that can be trusted: Root.Pem
A chain of untrusted certificates issued by the Root certificate: A.Pem, B.Pem, C.Pem
A private key file MyPrivate.key and a trusted certificate issued by C.Pemto ensure my identity: MyCertificate.pem
If needed, Convert certificate file to PEM format
If the certificates are not in PEM format, we need to convert them first. To check if they are in PEM, open them in a text editor. A PEM file looks like:
-----BEGIN CERTIFICATE-----
MIIFyjCCA7KgAwIBAgIEAJiWjDANBgkqhkiG9w0BAQsFADBaMQswCQYDVQQGEwJO
...
-----END CERTIFICATE-----
If it is not, we can use openSSL to convert the file:
openssl.exe x509 -inform DER -in myCertificate.cer -out myCertificate.Pem
inform: the format of the input file: DER / NET / PEM (well, if already PEM you won't have to convert)
in / out: the input file, the output file
Verify the certificate chain
For extra security, I verified every certificate separately. It is probably also safe to do this in one step.
Check validity of the root certificate. For instance by checking the fingerprint with a published fingerprint.
Check validity of the untrusted certificates
(1) Is A.pem issued by Root.Pem?
openssl.exe verify -show_chain -CAfile root.pem A.pem
Parameter -CAfile contains the trusted certificate. The last file is the file that contains the certificate to be verified.
Reply should be similar to:
A.pem: OK
Chain:
depth=0: C = NL, ..., CN = <some text describing certificate A> (untrusted)
depth=1: C = NL, ..., CN = <some text describing the trusted root certificate>
(2) Is B.Pem issued by the trusted A.Pem?
Now that A.pem can be trusted, we can check B.Pem. For this we mention the intermediate certificate A.Pem as untrusted as advised in this answer
openssl.exe verify -show_chain -CAfile root.pem -untrusted A.pem B.pem
Reply:
B.pem: OK
Chain:
depth=0: C = NL, ..., CN = <some text describing certificate B> (untrusted)
depth=1: C = NL, ..., CN = <some text describing certificate A> (untrusted)
depth=2: C = NL, ..., CN = <some text describing the trusted root certificate>
(3) Can we trust the rest of the certificate chain?
So now B can be trusted. To continue checking the chain, concatenate the untrusted CA-files into one untrusted.pem file. Do not add MyCertificate.Pem
-----BEGIN CERTIFICATE-----
MIIGNjCCBB6gAwIBA...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
jCCBB6gAwIBA34F..
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
dZBo31cAYsByRL...
-----END CERTIFICATE-----
And the command:
openssl.exe verify -show_chain -CAfile root.pem -untrusted untrusted.pem myCertificate.pem
Reply:
MyCertificate.pem: OK
Chain:
depth=0: C = NL, ..., CN = <some text describing MyCertificate> (untrusted)
depth=1: C = NL, ..., CN = <some text describing certificate C> (untrusted)
depth=2: C = NL, ..., CN = <some text describing certificate B> (untrusted)
depth=3: C = NL, ..., CN = <some text describing certificate A> (untrusted)
depth=4: C = NL, ..., CN = <some text describing the trusted root certificate>
I guess maybe all those intermediate steps were not necessary to check the validity.
Check Connection
Now that the certificate chain is trusted, we can use OpenSsl to check the connection.
Concatenate all certificates, except MyCertificate.pem in one file AllTrusted.pem, use a text editor, or a command Copy Root.Pem + A.Pem + B.Pem ... Trusted.Pem
Command:
openssl.exe s_client CAfile Trusted.Pem -connect google.nl:443
Replace google.nl:443 with the proper address and port
Reply, something similar to:
CONNECTED(00000124)
depth=1 C = US, O = Google Trust Services, CN = Google Internet Authority G3
verify error:num=20:unable to get local issuer certificate
---
Certificate chain
0 s:/C=US/ST=California/L=Mountain View/O=Google LLC/CN=google.com
i:/C=US/O=Google Trust Services/CN=Google Internet Authority G3
1 s:/C=US/O=Google Trust Services/CN=Google Internet Authority G3
i:/OU=GlobalSign Root CA - R2/O=GlobalSign/CN=GlobalSign
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIhWDCCIECgAwIBAgIQaEMB4EOx3++GhdWADJfgEjANBgkqhkiG9w0BAQsFADBU
...
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Mountain View/O=Google LLC/CN=google.com
issuer=/C=US/O=Google Trust Services/CN=Google Internet Authority G3
The server sent a certificate to identify itself. The client should use this certificate and its trusted CA-chain to check the identity of the server.
To continue communicating, we need a PEM file that contains the mentioned issuer and its issuers until the root. Use the procedure described above to get a complete certificate chain, and add all certificates in the correct order to a file trusted.pem. If you copy-paste the received certificate to a PEM file (text), you should be able to verify this received certificate the same way as I verified MyCertificate.Pem as described above.
Once the CA certificates for the received certificates were installed, my openssl s_client command replied with:
...
SSL handshake has read 8945 by
Verification: OK
---
New, TLSv1.2, Cipher is ...
Server public key is 2048 bit
Start Time: 1551779993
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
So the certificate chain to identify the server is accepted.
Idintify me to the server
The next step will be to check if I can identify myself at the server using MyCertficate.pem.
This is the first time I need my private key file. We'll use curl for this:
Command:
curl.exe -v --cacert trusted.pem --cert MyCertificate.pem --key MyPrivate.key https://...
-v: verbose
--cacert: the text file with the concatenation of the trusted CA chain until the root, as verified using openssl verify
--Cert: the certificate to be used by me to identify myself
--Key: the private key for this certificate
Reply:
...
* successfully set certificate verify locations:
* CAfile: trustall.pem
...
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
...
* Server certificate:
* subject: C=NL; ...
* start date: Apr 19 12:10:31 2016 GMT
* expire date: Apr 19 12:10:31 2019 GMT
...
* issuer: C=NL; O= <description of certificate issuer; should be in trusted.pem>
* SSL certificate verify ok.
> GET /exchange/ciot HTTP/1.1
> Host: ....
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 400 Bad Request
What we see:
TrustAll.Pem contains the trusted certificates
(Out) Client Hello - (In) Server Hello: apparently we are on speaking terms
Server sends certificate and requests one
Client sends its certificate to identify itself
Display of the received certificate, the one with which the server identifies itself. The issuer is expected to be in trusted.pem
The received certificate is verifies and accepted. Data transfer can start
Because I didn't send any data, the response is a 400 Bad Request
So this is enough to know that both client and server use trusted certificates and that communication is possible