I am trying to enable xpack in elasticsearch and followed the getting started blog post from elasticsearch site.
Things I did:
I ran this command $/usr/share/elasticsearch/elasticsearch-certutil cert -out config/elastic-certificates.p12 -pass "" and entered /etc/elasticsearch/elastic-certificates.p12 when asked for desired output file.
Edited the elasticsearch.yml config file located at /etc/elasticsearch/elasticsearch.yml and entered following lines:
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elastic-certificates.p12
Logs/ Exceptions from /var/log/elasticsearch/elasticsearch.log
2021-06-10T02:58:12,542][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested:
ElasticsearchException[failed to create trust manager]; nested: ElasticsearchException[failed to initialize SSL TrustManager - keystore file [/etc/el
asticsearch/elastic-certificates.p12] does not exist]; nested: AccessDeniedException[/etc/elasticsearch/elastic-certificates.p12];
Caused by: org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.transport.ssl]
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/elastic-certificates.p12
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/elastic-certificates.p12
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/elastic-certificates.p12
The permissions for my .p12 cert file is rw-------
What am i missing here?
I followed the documentation line by line
Your path in elasticsearch.yml is wrong.
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elastic-certificates.p12
If you placed the certificate in /etc/elasticsearch folder, then correct it.
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
Also make sure that owner of the cert is root:elasticsearch
sudo chown root:elasticsearch /etc/elasticsearch/elastic-c*
I got this exact same error because I had some leftover files and yes lines in the elasticsearch.yml file for security that were leftover from a previous run.
To rerun Elasticsearch (this worked for Elasticsearch for Windows 10, version 8.4.1) and eliminate this issue, go to the main Elasticsearch folder where it was installed then:
Delete the data and certs folders.
Delete the elasticsearch.keystore file.
Edit the elasticsearch.yml file by removing the generated lines and just leave the comments (default).
I also delete all the log files under the logs folder to clean things up.
Rerun the startup command for elasticsearch from a command prompt window: .\bin\elasticsearch.bat
Related
I can't get Elastic and Kibana to work with real certificates from either Let's encrypt or Comodo.
I've thought about it a thousand times, but no matter how much I read and reread, I can't find the error of something as simple as installing certificate chains. At first I thought it was a Let's Encrypt issue but this afternoon I bought some certificates issued by Comodo and nothing, the same.
/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
17:21:59.701 [main] WARN org.elasticsearch.common.ssl.DiagnosticTrustManager - failed to establish trust with server at [91.121.226.53]; the server provided a certificate with subject name [CN=elk2.mydomain.ovh], fingerprint [f611c9e9ce2462ed1f3c7e29035f2db2347e39df], keyUsage [digitalSignature, keyEncipherment] and extendedKeyUsage [serverAuth, clientAuth]; the session uses cipher suite [TLS_AES_256_GCM_SHA384] and protocol [TLSv1.3]; the certificate has subject alternative names [DNS:elk2.mydomain.ovh,DNS:www.elk2.mydomain.ovh]; the certificate is issued by [CN=Don Dominio / MrDomain RSA DV CA,O=Soluciones Corporativas IP\, SL,L=Manacor,ST=Illes Balears,C=ES] but the server did not provide a copy of the issuing certificate in the certificate chain; the issuing certificate with fingerprint [ce1710a80bd0dfb6ed688c37c8fc066268bd814b] is trusted in this ssl context ([xpack.security.http.ssl (with trust configuration: PEM-trust{/etc/elasticsearch/certs/elk2.mydomain.ovh.ca.crt})])
/etc/kibana/kibana.yml
server.host: "91.121.226.53"
server.publicBaseUrl: "https://elk2.mydomain.ovh:5601"
server.name: "elk2.mydomain.ovh"
server.ssl.enabled: true
server.ssl.certificate: certs/elk2.mydomain.ovh.crt
server.ssl.key: certs/elk2.mydomain.ovh.key
server.ssl.certificateAuthorities: ["certs/elk2.mydomain.ovh.ca.crt"]
elasticsearch.hosts: ["https://elk2.mydomain.ovh:9200"]
elasticsearch.ssl.certificate: certs/elk2.mydomain.ovh.crt
elasticsearch.ssl.key: certs/elk2.mydomain.ovh.key
elasticsearch.ssl.certificateAuthorities: [ "certs/elk2.mydomain.ovh.ca.crt" ]
elasticsearch.ssl.verificationMode: none
logging:
appenders:
file:
type: file
fileName: /var/log/kibana/kibana.log
layout:
type: json
root:
appenders:
- default
- file
pid.file: /run/kibana/kibana.pid
Elasticsearch
root#elk2:/etc/ssl# grep -Ev '^#|^$' /etc/elasticsearch/elasticsearch.yml
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
enabled: true
key: certs/elk2.mydomain.ovh.key
certificate: certs/elk2.mydomain.ovh.crt
certificate_authorities: ["certs/elk2.mydomain.ovh.ca.crt"]
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
key: certs/elk2.mydomain.ovh.key
certificate: certs/elk2.mydomain.ovh.crt
certificate_authorities: ["certs/elk2.mydomain.ovh.ca.crt"]
cluster.initial_master_nodes: ["elk2.mydomain.ovh"]
http.host: 0.0.0.0
Of course I have verified the files found in both the elasticsearch and kibana certs, just as I do with any software installation that requires certificates, including authority or CA
Way with LetsEncript
elasticsearch.yml
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
enabled: true
key: certs/privkey.pem
certificate: certs/fullchain.pem
certificate_authorities: [
"certs/chain.pem",
"certs/cacert.r3.pem"
]
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
key: certs/privkey.pem
certificate: certs/fullchain.pem
certificate_authorities: [
"certs/chain.pem",
"certs/cacert.r3.pem" ### Try also with pem root ISRG Root X1 cacert.x1.pem
]
cluster.initial_master_nodes: ["elk2.mydomain.ovh"]
http.host: 0.0.0.0
kibana.yml
server.host: "91.121.226.53"
server.publicBaseUrl: "https://elk2.mydomain.ovh:5601"
server.name: "elk2.mydomain.ovh"
server.ssl.enabled: true
server.ssl.certificate: certs/fullchain.pem
server.ssl.key: certs/key.pem
server.ssl.certificateAuthorities: [
"certs/chain.pem",
"certs/cacert.r3.pem"
]
elasticsearch.hosts: ["https://elk2.mydomain.ovh:9200"]
elasticsearch.ssl.certificate: certs/fullchain.pem
elasticsearch.ssl.key: certs/fullchain.key
elasticsearch.ssl.certificateAuthorities: [
"certs/chain.pem",
"certs/cacert.r3.pem"
]]
logging:
appenders:
file:
type: file
fileName: /var/log/kibana/kibana.log
layout:
type: json
root:
appenders:
- default
- file
pid.file: /run/kibana/kibana.pid
Error
/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
18:28:33.799 [main] WARN org.elasticsearch.common.ssl.DiagnosticTrustManager - failed to establish trust with server at [91.121.226.53]; the server provided a certificate with subject name [CN=elk2.endesarrollo.ovh], fingerprint [d3432f5a03043e533bd1a7a1aeabce8964bd1ac2], keyUsage [digitalSignature, keyEncipherment] and extendedKeyUsage [serverAuth, clientAuth]; the session uses cipher suite [TLS_AES_256_GCM_SHA384] and protocol [TLSv1.3]; the certificate has subject alternative names [DNS:elk2.endesarrollo.ovh]; the certificate is issued by [CN=R3,O=Let's Encrypt,C=US]; the certificate is signed by (subject [CN=R3,O=Let's Encrypt,C=US] fingerprint [a053375bfe84e8b748782c7cee15827a6af5a405] {trusted issuer}) signed by (subject [CN=ISRG Root X1,O=Internet Security Research Group,C=US] fingerprint [933c6ddee95c9c41a40f9f50493d82be03ad87bf] {trusted issuer}) which is issued by [CN=DST Root CA X3,O=Digital Signature Trust Co.] (but that issuer certificate was not provided in the chain); this ssl context ([xpack.security.http.ssl (with trust configuration: PEM-trust{/etc/elasticsearch/certs/chain.pem,/etc/elasticsearch/certs/cacert.r3.pem})]) is not configured to trust that issuer but trusts [2] other issuers ([CN=ISRG Root X1,O=Internet Security Research Group,C=US, CN=R3,O=Let's Encrypt,C=US])
java.security.cert.CertificateException: No subject alternative names matching IP address 91.121.226.53 found
```
Apreciate help.
I have a problem, I'm creating Elasticsearch as StatefulSet, and I need to use x-pack. For that, I know I need to configure the security properties below:
- name: xpack.license.self_generated.type
value: "basic"
- name: xpack.security.enabled
value: 'true'
- name: xpack.security.transport.ssl.enabled
value: 'true'
- name: xpack.security.transport.ssl.verification_mode
value: 'certificate'
- name: xpack.security.transport.ssl.keystore.path
value: '/usr/share/elasticsearch/elastic-certificates.p12'
- name: xpack.security.transport.ssl.truststore.path
value: '/usr/share/elasticsearch/elastic-certificates.p12'
About Transport TLS/SSL encryption, I know I should use "bin/elasticsearch-certutil ca" to generate the certificate.
What's my problem? when I apply YAML to my cluster, the certificate does not yet exist.
When I try to access bash, the pod is no longer available.
What would be the best strategy for this type of deployment?
*Error:
ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: ElasticsearchException[failed to initialize SSL TrustManager - access to read truststore file [/usr/share/elasticsearch/elastic-certificates.p12] is blocked; SSL resources should be placed in the [/usr/share/elasticsearch/config] directory]; nested: AccessControlException[access denied ("java.io.FilePermission" "/usr/share/elasticsearch/elastic-certificates.p12" "read")];
Likely root cause: java.security.AccessControlException: access denied ("java.io.FilePermission" "/usr/share/elasticsearch/elastic-certificates.p12" "read")
I hope it is not too late.
Apparently, the issue comes from the fact that the group and maybe the permission to the certificate file is different.
Check the certificate permission and group by ls -al. It should be as follows:
-rw-rw---- 1 root elasticsearch 3596 Mar 21 16:04 elastic-certificates.p12
-rw-rw---- 1 root elasticsearch 2672 Mar 21 16:04 elastic-stack-ca.p12
If it is different, use the following commands to fix the issue:
# change the group to `elasticsearch`
chgrp elasticsearch ./elastic-*.p12
# change the permission of the file to 660
chmod 660 ./elastic-*.p12
I'm following this post, security session to set up a new password for elastic user. Running ./bin/elasticsearch-setup-passwords interactive gave:
Failed to determine the health of the cluster running at http://es-ip-1:9200
Unexpected response code [503] from calling GET http://es-ip-1:9200/_cluster/health?pretty
Cause: master_not_discovered_exception
Unexpected response code [503] from calling PUT http://es-ip-1:9200/_security/user/apm_system/_password?pretty
Cause: Cluster state has not been recovered yet, cannot write to the [null] index
ERROR: Failed to set password for user [apm_system].
I entered the same password for all these elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user. What wrong did I do here, logging in elastic with new password doesn't work. Thanks.
Update
Config is the same on both nodes, except the network.host which is the private IP address of each node
elasticsearch.yml
network.host: es-ip-1
discovery.zen.ping.unicast.hosts: [es-ip-1, es-ip-2, es-ip-3]
discovery.zen.minimum_master_nodes: 2
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
Figured it out, gotta copy the cert to all the nodes, in this case it's elastic-certificates.p12.
I had this working on localhost about a month ago, but when I came back to the project, hitting the URL results in a "This site can’t provide a secure connection" error in the browser. For reference, I believe I had used this site as a guide to get it set up: https://www.thomasvitale.com/https-spring-boot-ssl-certificate/
I'm running the project with the bootRun Gradle task and the default embedded Tomcat server. The following configuration is set:
/grails-app/conf/application.yml
spring:
profiles: development
server:
port: 8092
ssl:
enabled: true
key-store: classpath:keystore.jks
key-store-password: password
key-store-type: pkcs12
key-alias: alias
key-password: password
The keystore is located at /src/main/resources/keystore.jks.
I don't see any errors in the console indicating any issues opening or accessing the keystore so I'm at a loss as to what might have changed. Is there a better guide than the one linked above that would give me other things to check for issues?
Eventually, I tried changing the password field to something I knew was not the password and the app started without errors. This led me to determine that the YAML file was not being picked up by bootRun. Removing the Spring profile config at the top of the YAML file then caused the app to error because of the incorrect password.
This led me to realize that somehow the active profile configuration in my Gradle bootRun task was reset. Adding -Dspring.profiles.active=development to the VM options fixes the problem.
I use k8s to deploy Elasticsearch.
Dockerfile:
FROM docker.elastic.co/elasticsearch/elasticsearch:6.3.1
USER elasticsearch
RUN elasticsearch-plugin install --batch analysis-kuromoji
RUN elasticsearch-plugin install --batch org.codelibs:elasticsearch-analysis-kuromoji-neologd:6.3.1
RUN elasticsearch-plugin install --batch com.floragunn:search-guard-6:6.3.1-22.3
RUN mkdir -p /usr/share/elasticsearch/batch/scripts
RUN mkdir -p /usr/share/elasticsearch/batch/logs
COPY searchguard_not_initialized_check_batch.sh /usr/share/elasticsearch/batch/scripts/
RUN rm -f /usr/share/elasticsearch/config/elasticsearch.yml && rm -f /usr/share/elasticsearch/plugins/search-guard-6/sgconfig/* && mkdir -p /usr/share/elasticsearch/data
File searchguard_not_initialized_check_batch.sh to run
sh /usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin.sh -diagnose -cd /usr/share/elasticsearch/plugins/search-guard-6/sgconfig -cn pnt-es.stg -key /usr/share/elasticsearch/config/cert/kirk-key.pem -cert /usr/share/elasticsearch/config/cert/kirk.pem -cacert /usr/share/elasticsearch/config/cert/root-ca.pem -nhnv
Elasticsearch.yml:
cluster.name: "pnt-es.stg"
discovery.zen.ping.unicast.hosts: elasticsearch-service
discovery.zen.minimum_master_nodes: {{ elasticsearch_log_minimum_master_nodes }}
network.host: ['_site_', '_local_']
node.name: ${HOSTNAME}
http.port: 9200
transport.tcp.port: 9300
path.data: /usr/share/elasticsearch/data
path.logs: /var/log/elasticsearch
searchguard.ssl.transport.pemcert_filepath: cert/esnode.pem
searchguard.ssl.transport.pemkey_filepath: cert/esnode-key.pem
searchguard.ssl.transport.pemtrustedcas_filepath: cert/root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: cert/esnode.pem
searchguard.ssl.http.pemkey_filepath: cert/esnode-key.pem
searchguard.ssl.http.pemtrustedcas_filepath: cert/root-ca.pem
searchguard.allow_unsafe_democertificates: true
searchguard.allow_default_init_sgindex: true
searchguard.enterprise_modules_enabled: false
searchguard.authcz.admin_dn:
- CN=kirk,OU=client,O=client,L=test,C=de
xpack.security.enabled: false
Error as below:
How can I fix it?
enter image description here
When I run this command, Error as below.
[elasticsearch#elasticsearch-daemonset-4qwcj ~]$ sh /usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin.sh --diagnos -cd /usr/share/elasticsearch/plugins/search-guard-6/sgconfig -cn pnt-es.stg -key /usr/share/elasticsearch/config/cert/kirk-key.pem -cert /usr/share/elasticsearch/config/cert/kirk.pem -cacert /usr/share/elasticsearch/config/cert/root-ca.pem -nhnv
Search Guard Admin v6
Will connect to localhost:9300 ... done
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/usr/share/elasticsearch/plugins/search-guard-6/netty-common-4.1.16.Final.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Elasticsearch Version: 6.3.1
Search Guard Version: 6.3.1-22.3
Connected as CN=kirk,OU=client,O=client,L=test,C=de
Diagnostic trace written to: /usr/share/elasticsearch/sgadmin_diag_trace_2019-Oct-03_05-35-51.txt
Contacting elasticsearch cluster 'pnt-es.stg' and wait for YELLOW clusterstate ...
vi /usr/share/elasticsearch/sgadmin_diag_trace_2019-Oct-03_05-35-51.Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ...
Root cause: MasterNotDiscoveredException[null] (org.elasticsearch.discovery.MasterNotDiscoveredException/org.elasticsearch.discovery.MasterNotDiscoveredException)
* Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
* Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
* If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
* Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ...
Root cause: MasterNotDiscoveredException[null] (org.elasticsearch.discovery.MasterNotDiscoveredException/org.elasticsearch.discovery.MasterNotDiscoveredException)
* Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
* Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
* If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
* Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ...
Root cause: MasterNotDiscoveredException[null] (org.elasticsearch.discovery.MasterNotDiscoveredException/org.elasticsearch.discovery.MasterNotDiscoveredException)
* Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
* Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
* If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
* Add --accept-red-cluster to allow sgadmin to operate on a red cluster.