I am new to ELK [ Elasticsearch Logstash and Kibana]. I installed Elasticsearch Logstash and Kibana in one server. Then installed Logstash in two machines. Total RAM in each system is around 30 GB. Total file to parse is around 300 GB. It took 6 days to filter out the searchd item[I searched for 10 digit number, timestamp and Name of one API from all these logs]and dispay it in Kibana. Am i doing something wrong here. Is there any other way to speed up this process.
Thanks in Advance,
Paul
You can filter out based on the time in Kibana UI. Also, if you are pushing to Logstash from any beat logger, Logstash takes time to push it to Elastic Search.
There are many beat applications which will directly push the data to Elastic Search.
Related
I just installed Elasticsearch and Kibana. X-pack has been enable. I never put any data into Elasticsearch, but when I use Kibana to monitor elasticsearch, I found the number of data increased automatically. I have to stop Elasticsearch and Kibana to make it stop. Why this phenomenon happen? And how can I fix it?
PS: Elasticsearch and Kibana version is 7.11.1.
The increasing number of data: kibana>stack monitoring>Elasticsearch>Overview>data.
The red part was from: 2M ---> 314M, I don't know what it done.
Sorry if this is a simple question - I'm new to ELK and have it all running with data coming through ok. My issue is that I'm concerned about storage growth given the number of records that will be coming through.
Having a search on the google I've seen that on GrayLog there is a setting to limit the amount of data to retain ( Graylog2- how to config logs retention to 1 week ) and I'd like to do the same in ELK but I can't find the correct setting.
There is no easy way to do this in GUI (yet). What you need is the Curator that can delete or rollup indices based on time (delete indices older than 7 days) or amount of documents in an index.
In a future Version there will be an inbuilt tool for that in Kibana, but it´s not in the current release (6.5). It will probably release with Elastic 6.6 (as a beta), but you may even have to wait for 7.X
Please, Can you give me yours suggestions about this problem ?
I have the indexed logs which arrive up to the cluster of elasticsearch, on the other hand, they do not appear in Kibana either in their TAGS or without their TAGS.
Thanks in Advance,
What are you exactly trying to do in Kibana? Let's say you are trying to make a histogram, then what are you using as the X-axis ?
I had a similar problem. My data was getting added to the Elasticsearch index but it was not getting visualized in kibana. It just said "no results found" no matter what I did. Then I discovered that I had a problem with the timestamp field that I was using as X-axis.
More specifically, the data arriving at the elasticsearch index had the current GMT+2 timestamp but kibana was configured with UTC time. So, kibana was configured to visualize data received in the last hours BUT the data that was arriving had a timestamp that was 2 hours AHEAD of the current time according to UTC.
I want to send queries to Elasticsearch through the Elasticsearch plugin within Logstash for every event in process. However, Logstash sends requests to Elasticsearch in bulk and indexed events are not immediately made available for search in Elasticsearch. It seems to me that there will be a lag (up to in process a second or more) between an index passing through Logstash and it being searchable. I don't know how to solve this.
Do you have any idea ?
Thank you for your time.
Joe
is it possible to show the size (physical size, e.g. MB) of one or more ES indices in Kibana?
Thanks
Kibana only:
It's not possible out of the box to view the disk-size of indices in Kibana.
Use the cat command to know how big your indices are (thats even possible without any Kibana).
If you need to view that data in Kibana index the output from the cat command to a dedicated Elasticsearch index and analyse it then in Kibana.
If other plugins/tools then Kibana are acceptable, read the following:
Check the Elasticsearch community plugins. The Head-Plugin (which I would recommand to you) gives you the info you want in addition to many other infos, like stats about your Shards, Nodes, etc...
Alternatively you could use the commerical Marvel Plugin from Elastic. I have never used it before, but it should be capeable of what you want, and much more. But Marvel is likely an overkill for what you want - so I wouldn't recommand that in the first place.
Although not a plugin of Kibana, cerebro is the official replacement of Kopf and runs as a standalone web server that can connect remotely to ElasticSearch instances. The UI is very informational and functional.
https://github.com/lmenezes/cerebro