Is there any easy way to configure it from Kibana? I have full control of the LogStash server and running Kibana on Elastic cloud
The only possible route I can think of is to create a custom button on Kibana that call a web service, setup a web service on LogStash server that manually start logstash on call
Is there any other way around this?
Related
When using logstash to retrieve airflow logs from a folder you have access to, would I still need to make any changes in the airflow.cfg file?
For instance, I have airflow and ELK deployed on same ec2 instance. The logstash .conf file has access to the airflow logs path since they are on the same instance. Do I need to turn on remote logging in airflow config?
In fact you have two options to push airflow logs to Elastic Search:
Using a log collector (logstash, fluentd, ...) to collect Airflow log then send it to Elastic Search server, in this case you don't need to change any Airflow config, you can just read the logs from the files or stdout and send it to ES.
Using Airflow remote logging feature, in this case Airflow will log directly to your remote logging server (ES in your case), and will store a local version of this log to show it when the remote server is unavailable.
So the answer to your question is no, if you have a logstash, you don't need Airflow remote logging config
Say, the elastic search cluster is secured via kerbros based authentication. So, how do I configure the jaeger collector and jaeger query to connect to the elastic search index?
Is there any way to copy all the data of an index from elasticsearch from my computer to cloud elastic ?
i'm working on localhost and now i want to migrate it to cloud.elastic.co
Cheers!
you can do this with a snapshot of your local cluster into s3, then a restore of that on Elastic Cloud.
Which cloud provider are you using? If it is AWS OpenSearch, it does not allow remote index operation from local ElasticSearch. Allows if it's an ElasticSearch in the AWS cloud and https is broadcasting.
If the data is critical, you can pull the data and send bulk requests to ElasticSearch in the cloud. I had to do so.
You can write your own application or you can send requests multi-threaded with a tool like Jmeter.
I'm new to ELK. We have a Spring Boot backend on a dedicated AWS instance. We have ELK stack on another instance (To the outside world only Kibana is available). Information gathering to ELK is carried out via Amazon SQS.
These information include logs and some business history about user (registration, any other action, etc).
In this case, I have a question. Is it possible to get back information by action, by user and use it in the backend responses?
I am guessing you want to use data present in Elasticsearch to be available to spring boot application. It is definitely possible. You will need to open up elasticsearch port on elasticsearch machine to specifically to EC2 instance on which spring boot is running. How to open port will depend on if they are on same vpc, different vpc, different aws account etc. Once port is open, you can either use Spring Data Elasticsearch or just rest calls to access elasticsearch api.
I am trying to understand how Kibana communicates with ElasticSearch. Does this communication happen b/w the browser and the ElasticSearch server directly (Bypassing the Kibana server) or all Kibana requests go through Kibana server which then pass them on to ElasticSearch.
I am trying to figure out how to implement security around my ElasticSearch & Kibana based servers.
In Kibana3, the browser would talk to elasticsearch directly. Any protection of elasticsearch had to be done via a front-end web server, typically nginx with a bunch of proxy rules to prevent certain actions, etc.
In Kibana4, they realized that having a server would be beneficial, so your browser talks to the kibana server, which talks to elasticsearch.
The "official" way to protect elasticsearch is with "shield", which handles authentication and authorization for actions against the server.
in kibana 5 a request is sent from a web browser to kibana backend(developped using hapi js) then this request will be sent to elasticsearch using elasticsearch plugin wich is one of the kibana core plugins.