I have a logstash server that somehow stopped listening on its syslog input (but didn't crash thats odd enough on itself but case for another question), it was configured to have a max queue of 100GB and, after some time (31gb of queue) i decided to restart it.
After restarting logstash it gets "stuck" on
[INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
and it doesnt start sending to Elasticsearch the messages on the queue (neither the ones on the queue nor newly received ones).
If i delete the queue folder and restart logstash then I get new events but obviously I lost all the old ones.
Why is logstash taking so long to process the persistent queue when its big? are there any settings i should tune to make the pipeline flow ?
I have saved the old Persistent Queue for looking deeper into it, any pointers?
I am trying to visualise Kafka logs through ELK stack. I particularly need to see number of messages unread by consumers in real-time. I have seen the log folder in Kafka but wasn't able to understand.
Where would I find information related to offsets to consumers and how do i upload it in elasticsearch ?
Is there a documentation about logs in kafka (e.g server.log, controller.log)
Log folder of Kafka doesn't hold consumer lag.
You'll want to export this data from consumer applications themselves or install external monitoring like Prometheus, Burrow, or Remora, then scrape and index into Elasticsearch
I am trying out the ELK to visualise my log file. I have tried different setups:
Logstash file input plugin https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html
Logstash Beats input plugin https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html with Filebeat Logstash output https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html
Filebeat Elasticsearch output https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html
Can someone list out their differences and when to use which setup? If it is not for here, please point me to the right place like Super User or DevOp or Server Fault.
1) To use logstash file input you need a logstash instance running on the machine from where you want to collect the logs, if the logs are on the same machine that you are already running logstash this is not a problem, but if the logs are on remote machines, a logstash instance is not always recommended because it needs more resources than filebeat.
2 and 3) For collecting logs on remote machines filebeat is recommended since it needs less resources than a logstash instance, you would use the logstash output if you want to parse your logs, add or remove fields or make some enrichment on your data, if you don't need to do anything like that you can use the elasticsearch output and send the data directly to elasticsearch.
This is the main difference, if your logs are on the same machine that you are running logstash, you can use the file input, if you need to collect logs from remote machines, you can use filebeat and send it to logstash if you want to make transformations on your data, or send directly to elasticsearch if you don't need to make transformations on your data.
Another advantage of using filebeat, even on the logstash machine, is that if your logstash instance is down, you won't lose any logs, filebeat will resend the events, using the file input you can lose events in some cases.
An additional point for large scale application is that if you have a lot of Beat (FileBeat, HeartBeat, MetricBeat...) instances, you would not want them altogether open connection and sending data directly to Elasticsearch instance at the same time.
Having too many concurrent indexing connections may result in a high bulk queue, bad responsiveness and timeouts. And for that reason in most cases, the common setup is to have Logstash placed between Beat instances and Elasticsearch to control the indexing.
And for larger scale system, the common setup is having a buffering message queue (Apache Kafka, Rabbit MQ or Redis) between Beats and Logstash for resilency to avoid congestion on Logstash during event spikes.
Figures are captured from Logz.io. They also have a good
article on this topic.
Not really familiar with (2).
But,
Logstash(1) is usually a good choice to take a content play around with it using input/output filters, match it to your analyzers, then send it to Elasticsearch.
Ex.
You point the Logstash to your MySql which takes a row modify the data (maybe do some math on it, then Concat some and cut out some words then send it to ElasticSearch as processed data).
As for Logbeat(2), it's a perfect choice to pick up an already processed data and pass it to elasticsearch.
Logstash (as the name clearly states) is mostly good for log files and stuff like that. usually you can do tiny changes to those.
Ex. I have some log files in my servers (incl errors, syslogs, process logs..)
Logstash listens to those files, automatically picks up new lines added to it and sends those to Elasticsearch.
Then you can filter some things in elasticsearch and find what's important to you.
p.s: logstash has a really good way of load balancing too many data to ES.
You can now use filebeat to send logs to elasticsearch directly or logstash (without a logstash agent, but still need a logstash server of course).
Main advantage is that logstash will allow you to custom parse each line of the logs...whereas filebeat alone will simply send the log and there is not much separation of fields.
Elasticsearch will still index and store the data.
I'm trying to visualize logs from my app. My logs formatted as json and stored in some file. I have filebeat installed which uses the same file as input. An filebeat could send the logs to Logstash and to Elasticsearch directly. Logstash could process logs, do something, parse them...
But my logs are json formatted already.
Elasticsearch are going to be installed on another server, another side of the planet...
so, my question is, Is there any good reason to use logstash in such scenario?( no need do any processing ), or is it ok to send logs to elasticsearch server directly?
I'm guessing the Logstash could do some buffering, but I want to keep my app's server light, don't want to install anything on top of it.
Thanks.
May this help you :https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html.
You can post the json into es by filebeat without Logstash, Logtstash is too heavy sometimes.
I am trying to set up EFK (elasticsearch, fluentd, kibana) on kubernetes cluster, so i used the following controller and service yaml files:
fluentd-es.yaml
https://github.com/kubernetes/kubernetes/blob/release-1.2/cluster/saltbase/salt/fluentd-es/fluentd-es.yaml
es-controller.yaml, es-service.yaml, kibana-controller.yaml and kibana-service.yaml
https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch
after running them, i had the following log output and kibana dashboard was unable to show me logs and charts (keep loading for ever like next image).
fluentd log snapshot:
elasticsearch log snapshot:
kibana log snapshot
You have two issues:
ES connection is gettng time out after retrying.So make sure you are defining right es config in fluentd.conf.
Also this is giving- BufferQueueLimitError which comes when your queue is filled due to connection time out. If you are expecting to fix this you should define:
buffered memory and
buffer_type memory
buffer_chunk_limit **m
buffer_queue_limit **
flush_interval ***s
disable_retry_limit false
retry_wait **s
refer-
https://docs.fluentd.org/v0.12/articles/buffer-plugin-overview#secondary-output
The logs are pretty much telling you .... there's a connection problem to Elasticsearch.