i have add elasticsearch & kibana yml files and few screenshots.for login kibana dashboard it take properly but kibana dashboard not appear it will redirect again back page.
video link - enter link description here
screenshot --> keycloak client configurations
elasticsearch.yml
# --------------------------------------------------------------------------------
# Enable security features
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["surangas-MacBook-Air.local"]
# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0
# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0
#----------------------- END SECURITY AUTO CONFIGURATION -------------------------
xpack.security.authc.realms.oidc.oidc1:
order: 0
rp.client_id: "kibana"
rp.response_type: code
rp.redirect_uri: "http://localhost:5601/app/home#/"
op.issuer: "http://localhost:8080/realms/oidc1"
op.authorization_endpoint: "http://localhost:8080/realms/oidc1/protocol/openid-connect/auth"
op.token_endpoint: "http://localhost:8080/realms/oidc1/protocol/openid-connect/token"
op.jwkset_path: "https://localhost:8080/realms/oidc1/protocol/openid-connect/certs"
op.userinfo_endpoint: "http://localhost:8080/realms/oidc1/protocol/openid-connect/userinfo"
op.endsession_endpoint: "http://localhost:8080/realms/oidc1/protocol/openid-connect/logout"
rp.post_logout_redirect_uri: "http://localhost:5601/security/logged_out"
claims.principal: email
claims.groups: "http://localhost:8080/claims/groups"
kibana.yml
this my kibana.yml file. here i have configure keycloak login page configurations
# =================== Search Autocomplete ===================
xpack.security.session.idleTimeout: "30m"
xpack.security.session.cleanupInterval: "1d"
xpack.security.authc.providers:
oidc.oidc1:
order: 0
realm: "oidc1"
description: "Keycloak"
basic.basic:
order: 1
# This section was automatically generated during setup.
elasticsearch.hosts: ['https://192.168.8.184:9200']
elasticsearch.serviceAccountToken: AAEAAWVsYXN0aWMva2liYW5hL2Vucm9sbC1wcm9jZXNzLXRva2VuLTE2NTUwMDc0NDM1MDY6YXRIcmpaVnRRclMwSHM4NmVJcWpVZw
elasticsearch.ssl.certificateAuthorities: [/Users/suranga/Desktop/Monitoring/test/keycloak/kibana-8.2.0/data/ca_1655007444485.crt]
xpack.fleet.outputs: [{id: fleet-default-output, name: default, is_default: true, is_default_monitoring: true, type: elasticsearch, hosts: ['https://192.168.8.184:9200'], ca_trusted_fingerprint: 82d6e3b36b6132052fb895809a97588fb366edf7f3dfba981e724194d2d19af3}]
Related
I have 2 elastic clusters (Cluster1 and Cluster2) and I am trying to configure a follower index in Cluster2 from a leader index of Cluster1.
I have followed the next steps:
Add Cluster1 as remote cluster in Cluster2.
Configuration image
Configure the next users:
In Cluster1 user "cross-cluster-user" with the role "remote-replication".
cross-cluster-user configuration image
In Cluster2 user "cross-cluster-user" with the role "remote-replication".
cross-cluster-user configuration image
When I try to create a follower index of "newblogs" index, I have the next error:
Can't create follower index no such index [newblogs]
index_not_found_exception: no such index [newblogs]
Error image
The newblogs index exists in Cluster1:
Get index result
My elasticsearch version is 8.3.3.
Any help will be appreciated.
Best regards.
elasticsearch.yml (Cluster1)
cluster.name: elastic-lab
node.name: ${HOSTNAME}
network.host: _eth1_
cluster.initial_master_nodes: ["node1"]
#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 28-08-2022 15:46:47
#
# --------------------------------------------------------------------------------
# Enable security features
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
#----------------------- END SECURITY AUTO CONFIGURATION -------------------------
elasticsearch.yml (Cluster2)
cluster.name: elastic-lab2
node.name: ${HOSTNAME}
network.host: _eth1_
cluster.initial_master_nodes: ["node1"]
#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 28-08-2022 16:07:28
#
# --------------------------------------------------------------------------------
# Enable security features
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
#----------------------- END SECURITY AUTO CONFIGURATION -------------------------
It was a very silly problem...I have 2 Clusters (Cluster1 and Cluster2) and each consists in 1 elasticsearch node (hostname: node1) and 1 kibana virtual machine (hostname: node4). "Node1" has different IP addresses in each cluster, but when I configured node1 (of Cluster1) as a seed node, the resolution of the name "node1" was the IP address of the node1 from Cluster2. This was the reason the remote cluster appeared as connected, it was connected to its own node1!
I have configured the seed node by IP (instead of hostname) and it seems to work! I had to change the option "verification_mode" in elasticsearch.yml of all nodes to the value "none" (because I was having SSL issues and this is only a lab).
Note: I didn´t have to configure any user or role for the clusters to connect, even though the documentation says so.
Best regards.
i already installed elastic and kibana 8.2 in ubuntu 22.04 and i try to access kibana from the browser of my host it told me "Kibana server is not ready yet"
this my elastic and kibana yml files :
network.host: 192.168.1.10
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# --------------------------------- Readiness ----------------------------------
#
# Enable an unauthenticated TCP readiness endpoint on localhost
#
#readiness.port: 9399
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false
#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 11-06-2022 21:39:47
# Enable security features
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["elastic"]
# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0
# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0
#----------------------- END SECURITY AUTO CONFIGURATION -------------------------
this for kibana yml :
server.host: "192.168.1.10"
# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""
# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# Defaults to `false`.
#server.rewriteBasePath: false
# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
#server.publicBaseUrl: ""
# The maximum payload size in bytes for incoming server requests.
#server.maxPayload: 1048576
# The Kibana server's name. This is used for display purposes.
#server.name: "your-hostname"
# =================== System: Kibana Server (Optional) ===================
# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key
# =================== System: Elasticsearch ===================
# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://192.168.1.10:9200"]
# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
i put the # ip of the ubuntu server in config file of kibana and elastic because i configured a static ip for the server
More probably, Kibana is no able to reach elasticsearch because you're providing a http link and not a https
elasticsearch.hosts: ["http://192.168.1.10:9200"]
However, ssl was enabled in the config file of elasticsearch
You can try to disable the ssl and the security features and then you can configure them one by one according to the status of your project.
# Enable security features
xpack.security.enabled: false
xpack.security.enrollment.enabled: false
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: false
keystore.path: certs/http.p12
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
enabled: false
I just did a fresh install of Elastic 8.2 and am now getting an error whenever I try to connect:
ConnectionError: other side closed - Local: 127.0.0.1:59648, Remote: 127.0.0.1:9200
I am using the Javascript library using the following code:
const elasticClient = new Client({
requestTimeout: 360000,
node: "http://localhost:9200",
auth: {
username: "elastic",
password: "xxxxxxxxxxxxx"
}
});
Here is my elasticsearch.yml:
# ======================== Elasticsearch Configuration =========================
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 0.0.0.0
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.type: single-node
#
#
#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 17-06-2022 16:14:15
#
# --------------------------------------------------------------------------------
# Enable security features
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
#cluster.initial_master_nodes: ["ip-172-31-24-223.us-east-2.compute.internal"]
# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
#http.host: 0.0.0.0
# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0
#----------------------- END SECURITY AUTO CONFIGURATION -------------------------
I was never having this issue on 8.1 before, what changed with 8.2 for this error to come up? Also just to note; the 8.1 version that was being used before was upgraded from 7.13.
I am trying to setup elasticsearch in my Windows system but when I am trying to run it its starting up and showing below reponse when I redirect to http://localhost:9200.
{
"name" : "DESKTOP-L8UKCFI",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "z8IfZcFaQfSti3P4jhZxbg",
"version" : {
"number" : "8.1.0",
"build_flavor" : "default",
"build_type" : "zip",
"build_hash" : "3700f7679f7d95e36da0b43762189bab189bc53a",
"build_date" : "2022-03-03T14:20:00.690422633Z",
"build_snapshot" : false,
"lucene_version" : "9.0.0",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}
But in console its showing something like this
[2022-03-16T11:26:12,307][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [DESKTOP-
L8UKCFI] received plaintext http traffic on an https channel, closing connection
Netty4HttpChannel{localAddress=/[0:0:0:0:0:0:0:1]:9200, remoteAddress=/[0:0:0:0:0:0:0:1]:5996}
[2022-03-16T11:31:56,806][WARN ]
[o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [DESKTOP-L8UKCFI] http
client did not trust this server's certificate, closing connection
Netty4HttpChannel{localAddress=/[0:0:0:0:0:0:0:1]:9200,
remoteAddress=/[0:0:0:0:0:0:0:1]:6215}
elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false
#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 16-03-2022 06:55:18
#
# --------------------------------------------------------------------------------
# Enable security features
xpack.security.enabled: false
xpack.security.enrollment.enabled: false
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: false
keystore.path: certs/http.p12
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
enabled: false
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["DESKTOP-L8UKCFI"]
# Allow HTTP API connections from localhost and local networks
# Connections are encrypted and require user authentication
http.host: [_local_, _site_]
# Allow other nodes to join the cluster from localhost and local networks
# Connections are encrypted and mutually authenticated
#transport.host: [_local_, _site_]
#----------------------- END SECURITY AUTO CONFIGURATION -------------------------
What does it mean someone let me know.
As of ES 8, SSL/TLS is ON by default for HTTP clients.
The WARN message says
http client did not trust this server's certificate
... which means that you need to tell your browser to trust the server certificate. it is self-signed by default, so that's probably the reason.
Or you can simply disable SSL in your elasticsearch.yml configuration, that would also work.
As #Val has already answered the question above just posting the code new users who wants to disable the SSL.
# --------------------------------------------------------------------------------
# Enable security features
xpack.security.enabled: false
xpack.security.enrollment.enabled: false
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: false
keystore.path: certs/http.p12
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
enabled: false
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
Add this to your environment variables:
- xpack.security.enabled=false
Full:
b-elastic:
image: docker.elastic.co/elasticsearch/elasticsearch:8.4.0-arm64
container_name: b-elastic
environment:
- discovery.type=single-node
- ES_JAVA_OPTS=-Xms750m -Xmx750m
- xpack.security.enabled=false
volumes:
- ./:/project
ports:
- 9200:9200
Another way is to simply run elasticsearch as
./elasticsearch -E xpack.security.enabled=false
It basically runs it with SSL disabled, allowing you to create HTTP connections with it.
http client did not trust this server's certificate, closing connection Netty4HttpChannel{localAddress=/[0:0:0:0:0:0:0:1]:9200, remoteAddress=/[0:0:0:0:0:0:0:1]:54479}
Simply means your browser is not trusting the software, so use https instead of http like https://localhost:9200/ it will work. I got this solution from internet
I have installed Filebeat deb package in Client-server(Linux Wind-River) and ELK in Elk-server(Ubuntu-16.04-server). The problem is, I can't receive logs from Client-server. I checked the network statistics and it seems 5044 port(Listening port) in ELK server is LISTENING. I can ping from both sides. I also have ssh connection in both directions.
This is the link which I used to install these packages on servers.
My Filebeat configurations:
filebeat.prospectors:
- type: log
# Change to true to enable this prospector configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths. paths:
- /var/log/filebeat/*
- /var/log/*.log
#- c:\programdata\elasticsearch\logs\*
document_type: log
#============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 3
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["192.168.10.3:5044"]
proxy_url: socks5://wwproxy.seln.ete.ericsson.se:808
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
# Certificate for SSL client authentication
ssl.certificate: "/etc/pki/tls/certs/logstash-forwarder.crt"
# Client Certificate Key
ssl.key: "/etc/pki/tls/private/logstash-forwarder.key"
I figured out the Error! The problem is the server IP in openssl.cnf should be the IP address of bridged Interface. And the certificate generated with this openssl.cnf should be used in both the servers. Further, I also shared the .key generated in ELK server to Client-server to be more secured/authenticate.