Firstly i do not like asking questions without code but I could not find any solution and need help about requesting big log datas from kibana and elasticsearch too.
I am trying to get 10000 documents from kibana on discover panel but it get me error. Error is : my 10000 documents are 5gb size and it just allows 2gb , I searched about spliting data but i could not do that on kibana.
i also tried
_msearch
it is not what i look for.
1- Do you guyz can tell me how can i do (if its possible) scroll on kibana - discover.
2- How can i get bigger datas from 2gb ?
if you can give me examples or link resources i will be so pleased.
Why do you want to search above 10k docs on the discover module?
If im undestanding well you want to download more than 2gb of data from Kibana, right? Put this on the Kibana.yml: xpack.reporting.csv.maxSizeBytes: 50971520
Related
Can I put the response result that I query in Kibana dev tools into elasticsearch directly?
Or must I write a script to achieve it?
Any recommends?
Ok So here is one basic understanding after discussion.
Please observe carefully.
If you have head plugin installed for ES .
search for .kibana index .
open the .kibana index and you will have all the designed dashboards listed there with processd info.
Think ES as another Storage from where you can read the data and put that data into Another ES index.
Refer to this link :
https://www.elastic.co/blog/kibana-under-the-hood-object-persistence
Tools you can opt is Logstash for Reading and writing.
Grok pattern learning can give you good lead about that.
Tell me if need some real time pics for same problem.
Happy learning.
It is like you cook in kitchen and ask to put the cooked food in kitchen again.If you cooked food better consume it :)
See the visualization or processed data you see on kibana end is just for kibana.The algorithms or processing techniques for the data set residing at elastic search will be applied over the upcoming data set.
So offcourse you can put/consume your data in Elastic search back again.
It depends what sort of requirement you are facing.
Note : Data in elastic search(inverted index) after kibana processing not gonna change its architecture, due to which you are able to apply another processing techniques from kibana over the same index assuming that data is in it's earlier state.
I am facing a problem with elastic search. I am using elasticsearch 5.6
When I am searching an index on some fields and I get to have more than 40000 results.
I found 2 problems:
When trying to access page 1001 (results 10001) I get an error and I understood I can increase the default 10,000, However I can accept this limitation and expose back to the user only the first 10,000 results.
When I am trying to sort by a specific field, the sort does not work. This is a huge problem for me as this search is used by a client UI and I must enable paging through the results. I read about the scroll API but I does not fit my requirements (user requests from UI).
Do you have any idea how to solve this problem?
Thank you.
Is it possible to see which are the most popular searched phrases/words within a particular index in elasticsearch.
Can this be set up in kibana at all.
You can do that by using Search Slow log - https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-slowlog.html
You can set the slow log setting dynamically too. Once this is set you should see the logs in index_search_slowlog.log. Ingest these logs back to elasticsearch and visualize in kibana. You can create the dashboard from this data.
We use these slow logs to monitor slow queries, popular queries etc.
is it possible to show the size (physical size, e.g. MB) of one or more ES indices in Kibana?
Thanks
Kibana only:
It's not possible out of the box to view the disk-size of indices in Kibana.
Use the cat command to know how big your indices are (thats even possible without any Kibana).
If you need to view that data in Kibana index the output from the cat command to a dedicated Elasticsearch index and analyse it then in Kibana.
If other plugins/tools then Kibana are acceptable, read the following:
Check the Elasticsearch community plugins. The Head-Plugin (which I would recommand to you) gives you the info you want in addition to many other infos, like stats about your Shards, Nodes, etc...
Alternatively you could use the commerical Marvel Plugin from Elastic. I have never used it before, but it should be capeable of what you want, and much more. But Marvel is likely an overkill for what you want - so I wouldn't recommand that in the first place.
Although not a plugin of Kibana, cerebro is the official replacement of Kopf and runs as a standalone web server that can connect remotely to ElasticSearch instances. The UI is very informational and functional.
https://github.com/lmenezes/cerebro
How do Elasticsearch and Solr compare in respect to the following:
Indexing logs.
Indexing events.
Indexing PDF documents.
Ease of creating and distributing visualizations. Kibana vs Banana.
Support and documentation for developers.
Any help is appreciated.
EDIT
More specifically, i am trying to figure out how exactly a PDF document or an event can be indexed at all. I have worked a little bit on Elasticsearch and since i am a fan of JSON, i found it quite useful when i tried to index structured data.
For example logs are mostly structured and thus i guess easier to index and search. Now what if i want to index the whole log file itself?
Follow up
Is Kibana the only visualization tool available for Elasticsearch?
Is Banana the only visualization tool available for Solr?
Here is an answer to try to address just the Elasticsearch aspect of the post.
Take a look at https://github.com/elastic/elasticsearch-mapper-attachments for handling PDFs
For events/logs, you would need to transform those into structured data to index in Elasticsearch. You can have a field in there for the source (the log file the data came from and other information like that) - you will have all the data in the whole log file indexed in that fashion. You can take advantage of ES aggregations to group results based on log file, calculate statistics, etc.
The ELK stack is definitely worth a look.
I don't know if Kibana is the only visualization tool but it is probably the most popular and likely to offer more than something else.