I have a problem, I'm creating Elasticsearch as StatefulSet, and I need to use x-pack. For that, I know I need to configure the security properties below:
- name: xpack.license.self_generated.type
value: "basic"
- name: xpack.security.enabled
value: 'true'
- name: xpack.security.transport.ssl.enabled
value: 'true'
- name: xpack.security.transport.ssl.verification_mode
value: 'certificate'
- name: xpack.security.transport.ssl.keystore.path
value: '/usr/share/elasticsearch/elastic-certificates.p12'
- name: xpack.security.transport.ssl.truststore.path
value: '/usr/share/elasticsearch/elastic-certificates.p12'
About Transport TLS/SSL encryption, I know I should use "bin/elasticsearch-certutil ca" to generate the certificate.
What's my problem? when I apply YAML to my cluster, the certificate does not yet exist.
When I try to access bash, the pod is no longer available.
What would be the best strategy for this type of deployment?
*Error:
ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: ElasticsearchException[failed to initialize SSL TrustManager - access to read truststore file [/usr/share/elasticsearch/elastic-certificates.p12] is blocked; SSL resources should be placed in the [/usr/share/elasticsearch/config] directory]; nested: AccessControlException[access denied ("java.io.FilePermission" "/usr/share/elasticsearch/elastic-certificates.p12" "read")];
Likely root cause: java.security.AccessControlException: access denied ("java.io.FilePermission" "/usr/share/elasticsearch/elastic-certificates.p12" "read")
I hope it is not too late.
Apparently, the issue comes from the fact that the group and maybe the permission to the certificate file is different.
Check the certificate permission and group by ls -al. It should be as follows:
-rw-rw---- 1 root elasticsearch 3596 Mar 21 16:04 elastic-certificates.p12
-rw-rw---- 1 root elasticsearch 2672 Mar 21 16:04 elastic-stack-ca.p12
If it is different, use the following commands to fix the issue:
# change the group to `elasticsearch`
chgrp elasticsearch ./elastic-*.p12
# change the permission of the file to 660
chmod 660 ./elastic-*.p12
Related
I am trying to enable xpack in elasticsearch and followed the getting started blog post from elasticsearch site.
Things I did:
I ran this command $/usr/share/elasticsearch/elasticsearch-certutil cert -out config/elastic-certificates.p12 -pass "" and entered /etc/elasticsearch/elastic-certificates.p12 when asked for desired output file.
Edited the elasticsearch.yml config file located at /etc/elasticsearch/elasticsearch.yml and entered following lines:
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elastic-certificates.p12
Logs/ Exceptions from /var/log/elasticsearch/elasticsearch.log
2021-06-10T02:58:12,542][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested:
ElasticsearchException[failed to create trust manager]; nested: ElasticsearchException[failed to initialize SSL TrustManager - keystore file [/etc/el
asticsearch/elastic-certificates.p12] does not exist]; nested: AccessDeniedException[/etc/elasticsearch/elastic-certificates.p12];
Caused by: org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.transport.ssl]
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/elastic-certificates.p12
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/elastic-certificates.p12
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/elastic-certificates.p12
The permissions for my .p12 cert file is rw-------
What am i missing here?
I followed the documentation line by line
Your path in elasticsearch.yml is wrong.
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elastic-certificates.p12
If you placed the certificate in /etc/elasticsearch folder, then correct it.
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
Also make sure that owner of the cert is root:elasticsearch
sudo chown root:elasticsearch /etc/elasticsearch/elastic-c*
I got this exact same error because I had some leftover files and yes lines in the elasticsearch.yml file for security that were leftover from a previous run.
To rerun Elasticsearch (this worked for Elasticsearch for Windows 10, version 8.4.1) and eliminate this issue, go to the main Elasticsearch folder where it was installed then:
Delete the data and certs folders.
Delete the elasticsearch.keystore file.
Edit the elasticsearch.yml file by removing the generated lines and just leave the comments (default).
I also delete all the log files under the logs folder to clean things up.
Rerun the startup command for elasticsearch from a command prompt window: .\bin\elasticsearch.bat
I use k8s to deploy Elasticsearch.
Dockerfile:
FROM docker.elastic.co/elasticsearch/elasticsearch:6.3.1
USER elasticsearch
RUN elasticsearch-plugin install --batch analysis-kuromoji
RUN elasticsearch-plugin install --batch org.codelibs:elasticsearch-analysis-kuromoji-neologd:6.3.1
RUN elasticsearch-plugin install --batch com.floragunn:search-guard-6:6.3.1-22.3
RUN mkdir -p /usr/share/elasticsearch/batch/scripts
RUN mkdir -p /usr/share/elasticsearch/batch/logs
COPY searchguard_not_initialized_check_batch.sh /usr/share/elasticsearch/batch/scripts/
RUN rm -f /usr/share/elasticsearch/config/elasticsearch.yml && rm -f /usr/share/elasticsearch/plugins/search-guard-6/sgconfig/* && mkdir -p /usr/share/elasticsearch/data
File searchguard_not_initialized_check_batch.sh to run
sh /usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin.sh -diagnose -cd /usr/share/elasticsearch/plugins/search-guard-6/sgconfig -cn pnt-es.stg -key /usr/share/elasticsearch/config/cert/kirk-key.pem -cert /usr/share/elasticsearch/config/cert/kirk.pem -cacert /usr/share/elasticsearch/config/cert/root-ca.pem -nhnv
Elasticsearch.yml:
cluster.name: "pnt-es.stg"
discovery.zen.ping.unicast.hosts: elasticsearch-service
discovery.zen.minimum_master_nodes: {{ elasticsearch_log_minimum_master_nodes }}
network.host: ['_site_', '_local_']
node.name: ${HOSTNAME}
http.port: 9200
transport.tcp.port: 9300
path.data: /usr/share/elasticsearch/data
path.logs: /var/log/elasticsearch
searchguard.ssl.transport.pemcert_filepath: cert/esnode.pem
searchguard.ssl.transport.pemkey_filepath: cert/esnode-key.pem
searchguard.ssl.transport.pemtrustedcas_filepath: cert/root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: cert/esnode.pem
searchguard.ssl.http.pemkey_filepath: cert/esnode-key.pem
searchguard.ssl.http.pemtrustedcas_filepath: cert/root-ca.pem
searchguard.allow_unsafe_democertificates: true
searchguard.allow_default_init_sgindex: true
searchguard.enterprise_modules_enabled: false
searchguard.authcz.admin_dn:
- CN=kirk,OU=client,O=client,L=test,C=de
xpack.security.enabled: false
Error as below:
How can I fix it?
enter image description here
When I run this command, Error as below.
[elasticsearch#elasticsearch-daemonset-4qwcj ~]$ sh /usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin.sh --diagnos -cd /usr/share/elasticsearch/plugins/search-guard-6/sgconfig -cn pnt-es.stg -key /usr/share/elasticsearch/config/cert/kirk-key.pem -cert /usr/share/elasticsearch/config/cert/kirk.pem -cacert /usr/share/elasticsearch/config/cert/root-ca.pem -nhnv
Search Guard Admin v6
Will connect to localhost:9300 ... done
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/usr/share/elasticsearch/plugins/search-guard-6/netty-common-4.1.16.Final.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Elasticsearch Version: 6.3.1
Search Guard Version: 6.3.1-22.3
Connected as CN=kirk,OU=client,O=client,L=test,C=de
Diagnostic trace written to: /usr/share/elasticsearch/sgadmin_diag_trace_2019-Oct-03_05-35-51.txt
Contacting elasticsearch cluster 'pnt-es.stg' and wait for YELLOW clusterstate ...
vi /usr/share/elasticsearch/sgadmin_diag_trace_2019-Oct-03_05-35-51.Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ...
Root cause: MasterNotDiscoveredException[null] (org.elasticsearch.discovery.MasterNotDiscoveredException/org.elasticsearch.discovery.MasterNotDiscoveredException)
* Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
* Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
* If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
* Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ...
Root cause: MasterNotDiscoveredException[null] (org.elasticsearch.discovery.MasterNotDiscoveredException/org.elasticsearch.discovery.MasterNotDiscoveredException)
* Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
* Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
* If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
* Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ...
Root cause: MasterNotDiscoveredException[null] (org.elasticsearch.discovery.MasterNotDiscoveredException/org.elasticsearch.discovery.MasterNotDiscoveredException)
* Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
* Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
* If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
* Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
remark: I am using win10.
My goal is when windows boot mongodb as a service with authentication start( you can not enter the database without authenticate) but I can not manage to do it on a windows machine ( in linux it worked)
I write here the steps I tried:
dowlnload MongoDB
change conf from default to the following
# mongod.conf
http://docs.mongodb.org/manual/reference/configuration-options/
# Where and how to store data.
storage:
dbPath: C:\MongoDB\Server\4.0\data
journal:
enabled: true
# where to write logging data.
systemLog:
destination: file
logAppend: true
path: C:\MongoDB\Server\4.0\log\mongod.log
# network interfaces
net:
port: 27017
bindIp: 127.0.0.1
security:
authorization: enabled
setParameter:
enableLocalhostAuthBypass: false
create a Admin user in the Admin collection.
db.createUser(
{
user: "....",
pwd: "...",
roles:
[
{ role: "root", db: "admin" }
]
}
)
Made it a service:
sc.exe create MongoDB
binPath=“\”C:\MongoDB\Server\4.0\bin\mongod.exe\”
–service
config=\”C:\MongoDB\Server\4.0\bin\mongod.cfg\”” DisplayName= “MongoDB” start= “auto”
getting feedback Successful.
but when i restart the computer, mongod is not starting and if i dont specify mongod --auth i can still enter without a authentication
How can I run Mongod as service with authentication? what am i doing wrong?
When i am trying to activate the service manually I get the following error
Error photo
The issue with the security tag. I have the same issue when I wanted to start the service in Windows 10. I copy the command from Windows service properties and then run on the command prompt.
The prompt shows me the error:
Unrecognized category : security
I found the solution and it is to write the security tag with options properly.
YAML need some specific input I guess. Here it is the solution.
security:
authorization: enabled
I had the same issue.
In your mongodb.cfg, use 2 spaces (instead of TAB) to indent authorization: enabled
I am having a little bit of challenge with NiFi…MiNiFi precisely. We use MiNiFi to ship logs from remote systems to a NiFi Instance, from there to Kafka and into Elasticsearch. We can successfully do this without https, However, recently I was tasked to do same securely using https.
Using certificates, I can connect to the NiFi UI, the challenge is that MiNiFi is unable to connect to the RPG on the remote NiFi with the error "Unable to communicate with Remote NiFi at URI https://xxxx.com:9443/nifi due to: Received fatal alert: handshake_failure" . I suspect this is because of the errors below
2018-07-23 16:27:23,083 INFO [main] o.apache.nifi.controller.FlowController Not enabling RAW Socket Site-to-Site functionality because nifi.remote.input.socket.port is not set
2018-07-23 16:27:23,083 INFO [main] o.apache.nifi.controller.FlowController Not enabling HTTP(S) Site-to-Site functionality because the 'nifi.remote.input.http.enabled' property is not true
I have tried to set these properties in the nifi.properties file of MiNiFi, but the file is always overwritten at each restart with default values loaded.
Please, do you have any ideas on how to resolve this?
How can I bootstrap these settings at startup in the config.yml file or any other place?
You'll need to set those in the original flow that you export from NiFi to MiNiFi. The nifi.properties of the MiNiFi instance is automatically generated from the provided config.yml file. That file is generated by using the MiNiFi Converter Toolkit to convert the exported template XML file.
For more, you can watch these videos or read the Getting Started Guide.
You'll want to look for lines like the following in the config.yml:
Security Properties:
keystore: /tmp/ssl/localhost-ks.jks
keystore type: JKS
keystore password: localtest
key password: localtest
truststore: /tmp/ssl/localhost-ts.jks
truststore type: JKS
truststore password: localtest
ssl protocol: TLS
Sensitive Props:
key:
algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
provider: BC
Remote Processing Groups:
- name: http://localhost:8080/nifi
url: http://localhost:8080/nifi
comment: ''
timeout: 30 sec
yield period: 10 sec
Input Ports:
- id: AUTOGENERATED_NIFI_PORT_ID_HERE
name: MiNiFi-input
comment: ''
max concurrent tasks: 1
use compression: false
Properties: # Deviates from spec and will later be removed when this is autonegotiated
Port: 1026
Host Name: localhost
I installed elastic search using the deb package from here. The service however fails to start and throws below error. How can I fix this?
$ sudo service elasticsearch restart
* Stopping Elasticsearch Server [ OK ]
* Starting Elasticsearch Server
chown: invalid group: `elasticsearch:elasticsearch'
chown: invalid group: `elasticsearch:elasticsearch' [fail]
Maybe you don't have this user/group set up in your system:
For security reasons, running the server as an unprivileged user and group is strongly encouraged. Create user and group for Elasticsearch:
groupadd elasticsearch
useradd -s /sbin/nologin -d /usr/local/elasticsearch -c "Elasticsearch User"