I locally installed Elasticsearch with X-Pack and Kibana 8.2.2 and made it up and running. I also created a new project with Spring and Java that would connect with Elasticsearch. When running Elasticsearch for the first time it creates a certificate for the http and transport layers.
I want to use a certificate in Spring to login into Elasticsearch. Therefore I added elasticsearch-rest-client. My problem is, which certificate do I need from Elasticsearch to connect to it and be able to create e.g. role-mappings? So that I have superuser rights?
Quoting the documentation:
When Elasticsearch is configured to require client TLS authentication, for example when a PKI realm is configured, the client needs to provide a client certificate during the TLS handshake in order to authenticate. The following is an example of setting up the client for TLS authentication with a certificate and a private key that are stored in a PKCS#12 keystore.
If the client certificate and key are not available in a keystore but rather as PEM encoded files, you cannot use them directly to build an SSLContext. You must rely on external libraries to parse the PEM key into a PrivateKey instance. Alternatively, you can use external tools to build a keystore from your PEM files, as shown in the following example:
openssl pkcs12 -export -in client.crt -inkey private_key.pem -name "client" -out client.p12
How to set up PKI user authentication has a multi-page documentation and you'll be best served by using the one for your specific Elasticsearch version. The current version is available at https://www.elastic.co/guide/en/elasticsearch/reference/current/pki-realm.html but replace the current in the URL with whatever minor version you want to use (like 8.6).
PS: PKI authentication is a platinum licensed feature. But you can of course set it up with the free 30 day trial to see how it works.
Related
THIS IS STILL AN ISSUE ANY HELP WOULD BE APPRETIATED
I am having an issue setting up TLS through a custom CA between Kibana and Enterprise search. I have the default x-pack security set up for the interconnection of my Elasticsearch nodes with both Kibana and Enterprise search, which was done according to the following docs: minimal security basic security ssl/tls config. I can successfully run Enterprise search through http, however my issue arises when I enable ssl/tls for ent-search..
When I have https configured for ent-search using this doc, the server is "running", however I receive an error after boot and Kibana throws an error when attempting to connect.
ent-search error (non corresponding with Kibana's hit to the ent-search hostname, this error raises shortly after ent-search is "starting successfully", but isn't fatal)
[2022-06-14T20:37:45.734+00:00][6081][4496][cron-Work::Cron::SendTelemetry][ERROR]: Exception:
Exception while performing Work::Cron::SendTelemetry.perform()!: Faraday::ClientError: PKIX path
building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid
certification path to requested target
Kibana error (directly corresponding to when I ping http://obfuscated-dns:5601/app/enterprise_search/overview)
[2022-06-14T20:43:51.772+00:00][ERROR][plugins.enterpriseSearch] Could not perform access check to
Enterprise Search: FetchError: request to https://obfuscated-dns:3002/api/ent/v2/internal/client_config
failed, reason: unable to get issuer certificate
The steps I took to generate said certificate were: I created a csr on my server using elasticsearch-certutil csr along with a yml file which specified the distinguished name, I sent the unzipped csr to my CA (Digicert), uploaded the signed certificate and intermediate certificate provided by Digicert to my server, used openssl to generate a keystore using the signed cert and that private key generated along-side the original csr, then finally converted the keystore to .jks format using keytool.
From my understanding, the path of this keystore is what is needed for the configuration file for enterprise-search and the intermediary cert is what is used in the Kibana certificate authority config field (ca.pem). I have also tried to stuff both the signed and intermediate cert in the same .pem, as well as the private-key, signed and intermediate cert. Below are the relevant configurations:
kibana.yml
enterpriseSearch.host: https://ofuscated-dns:3002
enterpriseSearch.ssl.verificationMode: certificate
enterpriseSearch.ssl.certificateAuthorities:
- /path/ca.pem
enterprise-search.yml
ent_search.external_url: https://obfuscated-dns:3002
ent_search.listen_host: 0.0.0.0
ent_search.listen_port: 3002
ent_search.ssl.enabled: true
ent_search.ssl.keystore.path: "/path/keystore.jks"
ent_search.ssl.keystore.password: "pass"
ent_search.ssl.keystore.key_password: "pass"
I'm starting to feel like I fundamentally misunderstand something here. A lot of the jargon behind SSL/TLS certificates seems to lack standardization. While we are at it, what is a root cert in relation to what I have listed? Is it the intermediate cert? I see there is a master "root certificate" for the Digicert CN I certified under, however I'm unsure where this fits in. The config variable "certificateAuthorities" doesn't document what this .pem file should contain specifically and when searched the concept of a certificate authority is never associated with file contents, but instead is simply abstracted to the entity which provides certification (duh).
To put it succinctly: What does this variable "certificateAuthorities" explicitly entail?
UPDATE 09/28/2022
I have now confirmed that SSL is working when calling enterprise-search outside of the VM its running in. I can utilize its endpoint with my flutter and react app, however Kibana is till throwing the error mentioned above. I have checked that the root/intermediate CA provided to kibana's configuration is indeed the certificate linked with the signed cert provided to enterprise search and even confirmed so using SSLPoke.. This leaves me with the suspicion that perhaps Java is a bad actor in the mix? I've added the root/intermediate CA to the cacerts keystore in the ssl/java directory of the Linux VM, but still no luck. Any thoughts?
Note: My database works because I am able to migrate and query from within the application.
So I've setup my application and database on digital ocean's new "App" feature. And now I am trying to connect to the production database but I'm getting connection timeout error. Below are the details I inputed, but maybe I am missing my SSL cert ? If yes, how do I go about getting this ? Because I believe digital ocean automatically setup the SSL for me..
Database : postgresql
Tool used to connect : TablePlus
UPDATE
My site is SSL'ed
Checked directly on DO, and there's no certificate stored on my account
So after a few days I gave up and swap to using Droplets. I feel Apps Platform is cool for quick prototypes but I don't have much control in what I want can do in it. I don't think I could even run sudo in the console when I tried.
Here are a few other links I ran through that may or may not help others that was in similar situation if you were using a dev database on digital ocean. And also my support tickets solutions that I received.
Links
https://www.digitalocean.com/community/questions/cannot-connect-with-dev-database-due-to-ssl-issue?answer=67513
https://docs.digitalocean.com/products/databases/postgresql/how-to/connect
How to add an SSL certificate (ca-cert) to node.js environment variables in order to connect to Digital Ocean Postgres Managed Database?
DO Support Replied
I understand that you are trying to connect to the database with an
SSL certificate. Firstly, I want to let you know that you have
attached the dev database to your app "mysite" and not the production
database. However, you should be able to add the following env
variable to store the SSL certificate:
KEYS: CA_CERT VALUES: ${mysitedb.CA_CERT}
Once you add the above env variable then you should be able to use the
"CA_CERT" variable to fetch the SSL certificate in your app.
Additionally, you can view the SSL certificate by running the
following command in the console:
echo $CA_CERT
Thanks for getting back to us. This output would be intended. To use
the contents of that cert you would need to save the env variable to a
file or convert it from string using a method within your application.
You can then specify that file in your configuration and use the
certificate in your connection to the database.
An example of how to do this can be found here:
How to add an SSL certificate (ca-cert) to node.js environment variables in order to connect to Digital Ocean Postgres Managed Database?
Let us know if you have any questions.
Thank you for getting back to us!
From the screenshot, I see you are using the incorrect port number
5432. You have to use port number 25060 to connect to the database.
Regarding the SSL certificate, one thing to note here is that the
database is managed by DigitalOcean and it is not possible to generate
clients key (private key) and certificate (public key) via cloud
panel. That is the reason, you will need to generate those on your
local machine or from whichever client you plan to establish a
connection to the database.
Here is an example of how to use SSL on a client like Navicat, you
will need to download or have OpenSSL installed on your operating
system which you will use to generate Client Key File and Client
Certificate File which will be referred to as private and public keys
respectively
You need the below three files to connect
Client Key File
Client Certificate File
CA Certificate File
https://www2.navicat.com/manual/online_manual/en/navicat/linux_manual/SSLSettings.html
As a guide to establishing a connection
Generate Client Key File and Client Certificate File from your client or the local machine
Copy the certificate from app console to any .crt file and pass that file to connect to database.
Here is how to use the OpenSSL to generate the Client Key File and
Client Certificate File:
https://knowledge.digicert.com/solution/SO27347.html
openssl req -x509 -newkey rsa:2048 -keyout client-key.pem -out
client-cert.pem -days 3650 -nodes -subj '/CN=localhost'
For the Client Key File, Client Certificate, and CA Certificate choose
the directory location you saved them and click on "Test" button in
your client to test the connection.
Please let us know if you have any additional questions, and have a
wonderful day!
Could you please help me setup the SSL on the Nifi Application.
To explain about the steps taken so far.
I have used the following link intructions to use the CA signed certs provided to us (This include root,intermediate and Server cert). I have sucessfully configured Nifi to run on SSL on server end but i am not getting the steps to create a client cert so that using the client cert we can login to Nifi.
Help in this regard will be highly appreciated.
You'll need to generate a Certificate Signing Request (CSR) or request from your security/IT team who provided the CA-signed server certs that they provide a client certificate (and private key) signed by the same intermediate or root CA. You could also generate your own client certificate signed by a self-signed CA and put the public certificate of that CA in the NiFi truststore. More documentation around this process can be found in the NiFi Toolkit Guide.
When an application optionally allows you to specify a certificate which represents a certificate authority, what is that doing?
Does that basically set that cert as a "trusted" certificate?
Specific example:
Kibana communicates with elasticsearch. When you configure kibana, you can set the following configuration value:
elasticsearch.ssl.ca
The documentation says this value is an "Optional setting that enables you to specify a path to the PEM file for the certificate authority for your Elasticsearch instance." (source)
Within that same config you're also specifying a certificate and key that can be used for communication with the elasticsearch instance.
If the setting is optional it means that the default behaviour is to use the system Root CAs to validate the SSL Server certificate used by your Elasticsearch instance. If you used a standard commercial SSL Server certificate this should be enough.
Yes, you are setting that certificate as a Root CA but only for this application.
Using the setting you can specify the Root CA used to generated the SSL Server certificate. This is useful if you:
use a selfsigned certificate
use a Root CA that is not available in the system Root CA repository
need to have stringent security settings limiting subset of Root CAs trusted by your application.
I use a Java client using JMS/JNDI. The connection is working when using one-way SSL authentication SSLCAUTH(OPTIONAL). The first step I did is to export the public certificate of the client:
keytool -export -keystore keystore -storepass storepass -alias CLIENT -file client.cer
Then I added this certificate to the MQ key repository.
gsk7cmd -cert -add -db keydb.kdb -pw password -label ibmwebspheremqclient -file client.cer -format binary
And I finally switched to SSLCAUTH(REQUIRED) mode.
I get the following error log. The message is pretty clear, it can't find my client certificate. I read that my client certificate should have a label ibmwebspheremq<client_user_id>. What is this user id since I am connecting via Java?
AMQ9637: Channel is lacking a certificate.
As per the Technote Specifying the userid in the SSL certificate label for an MQ client the Java and JMS clients do not find their certificate based on the label.
What is often the problem is a mis-match between the trust store and the key store. I have seen two problems fairly commonly.
The application specifies a trust store but not a keystore. This works great for anonymous (one-way) SSL but not for mutual authenticated SSL. The app must specify both key store and trust store for mutual auth.
Sometimes the app specifies a trust store file but the private certs are in the key store. Or the app specifies the same file for both trust store and the key store and the personal certs are actually in a separate trust store file.
Does either of these solve the problem? If not, please update the question with a keytool -cert -list for both the key store and the trust store and the part of the command line or code that sets up the keystore/truststore.