I've deployed EFK stack in IBM Kuberentes cloud by following the step by step guide from this article. Every deployment is done successfully, all EFK stack are deployed fine, but I'm unable to find fluentd indices in elastic search,
I'm unable to create index pattern in kibana for log-data
Related
I've deployed grafana, loki and promtail in my local kubernetes cluster via the grafana helm charts
In order to improve the log output I want to configure derived fields for the Loki Data Source in Grafana.
But I would like to provide these configuration automatically via my helm deployment.
I've found no such properties in the grafana, loki or promtail charts.
Spring Cloud Sleuth is used for creating traceIds (Unique to request across services) and spanId (Same for one unit for work). My idea is that Zipkin server is used to get collective visualization of these logs across service. But I know and have used ELK stack which does necessarily the same function. I mean we can group requests with the same traceId for visualising, using ELK stack. But I do see people trying to implement distributed tracing with Sleuth, ELK along with Zipkin, as in these examples (Link1,Link2). But why do we need Zipkin if there is already ELK for log collection and visualising? Where I am missing?
Is there a way to run Elastic cloud on Kubernetes (ECK) with helm3?
As much as i know, there is no helm chart for ECK operator however for the Elasticsearch stack there is a helm chart available.
Elastic stack helm chart : https://github.com/elastic/helm-charts/tree/master/elasticsearch
ECK is operate you can extend Kubernetes orchestration by YAML files or else you can create own helm chart as per need if required.
ECK quick deploy : https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-quickstart.html
ECK is providing official support in GKE however i think you have idea about the advantages & disadvantages of using ECK.
Yes, starting ECK 1.3.0, there is an official Helm chart for deploying the operator.
I have installed Prometheus into a Kubernetes cluster using the helm stable chart. We run Elastic Search and I want to scrape metrics from this and then create Alerts based on events.
I have installed the elasticsearch exporter via helm but no where can I find how I then import these metrics into Prometheus ?
There is some config I am missing such as creating a scraping job or something. Anyone can help much appreciated.
I connected to the elasticsearch exporter and can see it pulling metrics.
If you're using an elasticsearch exporter it should contain some documentation. There are more than just one solution out there and you didn't specify which one you're using. In my opinion it would be best for you to start from a tutorial like this one which explains step by step the whole process. As you can read there:
Metrics collection of Prometheus follows the pull model. That means,
Prometheus is responsible for getting metrics from the services that
it monitors. This process introduced as scraping. Prometheus server
scrapes the defined service endpoints, collect the matrixes and store
in local database.
which means you need to configure Prometheus to scrape metrics exposed by the elasticsearch exporter you chose.
Official Prometheus documentation will be also a great source of knowledge and good starting point.
EDIT:
If you run your Elasticsearch instance on Kubernetes cluster, you should rather use the Service Discovery mechanism than static configs. More on <kubernetes_sd_config> you can find here.
There are five different types of Kubernetes service discoveries you can use with Prometheus: node, endpoints, service, pod, and ingress. The one which you most probably need in your case is endpoints. Prometheus uses the Kubernetes API to discover targets. Below you have some examples:
https://blog.sebastian-daschner.com/entries/prometheus-kubernetes-discovery
https://raw.githubusercontent.com/prometheus/prometheus/master/documentation/examples/prometheus-kubernetes.yml
We are going to deploy an Elasticsearch in a VM and configure our Logstash output to point to it. We don't plan for a multiple node cluster or cloud for hosting Elasticsearch. But we are checking for any possibility to fallback to our system-locally run Elasticsearch service, in case of any connection failure to the VM hosted Elasticsearch.
Is it possible to configure in Logstash in any way to have such fallback, in case connection to Elasticsearch is not available.
We use 5.6.5 version of Logstash and Elasticsearch. Thanks!