I'm pretty well versed in Splunk, and am trying to pickup ELK.
I have an instance up and running, but I am struggling to build a mental map of ELK (likely due to my experiment with Splunk)
Is there a 'stats' like command in ELK, where I could say something like
* | stats count by Variable
or even better
* | stats p50(Variable)
what would those commands be? (or is my mental model incorrect?)
I don't know Splunk so I can't really tell what stats means.
But I guess you want to run here an elasticsearch aggregation which looks like:
GET test/_search
{
"aggs": {
"my_stats": {
"stats": {
"field": "variable"
}
},
"my_p50": {
"percentiles": {
"field": "variable"
}
}
}
}
Note that this is to run on elasticsearch. Kibana proposes some visualizations to do the same but using the frontend. There is no CLI in Kibana.
You can run the query I pasted in Kibana Console available at http://0.0.0.0:5601/app/kibana#/dev_tools/console if you are using elastic stack 5.0.
Related
I want to carry out a customized aggregation through mapping a certain field in ES doc.
I try to leverage terms aggregation script. There're hundreds of mappings so that I have to put all of them into a HashMap:
GET /myindex/_search
{
"query": {
"match_all": {}
},
"aggs": {
"myagg": {
"terms": {
"script": {
"source": "Map m = new HashMap(); m.put('a', 'A'); m.put('b', 'A'); m.put('bb', 'CC'); ... return m.get(doc['foo.keyword'].value)",
"lang": "painless"
}
}
}
},
"size": 0
}
It's ugly and the performance is so bad even though I use the stored script. Have no idea why executing costs so much time in my script.
I also tried scripted metric aggregation. It's better but still slow compared with normal terms aggregation.
Is there any way to accelerate the mapping? (except runtime fields as My ES version does not support it)
tldr you could try runtime fields if you had them, but chances are they will also not be fast. that's the unfortunate nature of scripting in Elasticsearch at this point
your best bet would be what ExplodZe says above, and do that work prior during the ingestion process, to make query time faster
To display recent exceptions on a Grafana dashboard I am doing a query on exceptions in logfiles. Grafana doesn't seem to have an option to limit a string value in table view. Of course the stacktraces are huge.
So I came up with the idea to limit this field in the used Lucene query, but I am unaware on how to do this. I tried doing this using a painless script:
{
"query": {
"match_all": {}
},
"script_fields": {
"message_short": {
"script": {
"lang": "painless",
"inline": "return doc['message'].value.substring(50);"
}
}
}
}
I don't get any error but also no additional field "message_short" which I would have expected. Do I have to enable scripting support somehow? I'm running on v6.1.2
I got a workaround implemented where I have a drilldown URL ("Render value as link" in Grafana Table) where I render a link to my Kibana instance and use the Grafana variable $__cell that references the document_id I get from the underlying Elasticsearch query:
https://mykibana.host/app/kibana#/doc/myindex-*/myindex-prod-*/logs?id=$__cell&_g=h#8b5b71a
Not perfect, but keeps my Dashboard readable and allows more info if needed. Even better would be to add a shorted field into the ES index, but that is not possible for me currently.
I would like to run the following aggregation query in Kibana:
GET _search
{
"size": 0,
"aggs": {
"group_by_host": {
"terms": {
"field": "host",
"size": 20
}
}
}
}
I can run it in the dev tools console (what used to be called Sense), but I would like to run it in the Kibana proper. Having a hard time figuring it out.
Just create a Chart from Visualize tab.
Then buckets => X Axis (or Split Rows or whatever based on your chart type) => Terms => Choose your field.
Then click Advanced link and write {"size":10} to there:
Hope that helps!
I am using elasticsearch-dsl for my python application to query elastic search.
To debug what query is actually getting generated by elasticsearch-dsl library, I am unable to log or print the final query that goes to elasticsearch.
For example, like to see the request body sent to elasticsearch like this :
{
"query": {
"query_string": {
"query": "Dav*",
"fields": ["name", "short_code"],
"analyze_wildcard": true
}
}
}
Tried to bring the elasticsearch log level to TRACE. Even then, unable to see the queries that got executed.
Take a look at my blog post here, "Slowlog settings at index level" section. Basically, you can use slowlog to print in a separate log file Elasticsearch generates, the queries. I suggest using a very low threshold to be able to see all the queries.
For example, something like this, for a specific index:
PUT /test_index/_settings
{
"index": {
"search.slowlog.level": "trace",
"search.slowlog.threshold.query.trace": "1ms"
}
}
Or
PUT /_settings
{
"index": {
"search.slowlog.level": "trace",
"search.slowlog.threshold.query.trace": "1ms"
}
}
as a cluster-wide setting, for all the indices.
And the queries will be logged in your /logs location, a file called [CLUSTER_NAME]_index_search_slowlog.log.
I have documents which contains only "url"(analyzed) and "respsize"(not_analyzed) fields at first. I want to update documents that match the url and add new field "category"
I mean;
at first doc1:
{
"url":"http://stackoverflow.com/users/4005632/mehmet-yener-yilmaz",
"respsize":"500"
}
I have an external data and I know "stackoverflow.com" belongs to category 10,
And I need to update the doc, and make it like:
{
"url":"http://stackoverflow.com/users/4005632/mehmet-yener-yilmaz",
"respsize":"500",
"category":"10"
}
Of course I will do this all documents which url fields has "stackoverflow.com"
and I need the update each doc oly once.. Because category data of url is not changeable, no need to update again.
I need to use _update api with _version number to check it but cant compose the dsl query.
EDIT
I run this and looks works fine:
But documents not changed..
Although query result looks true, new field not added to docs, need refresh or etc?
You could use the update by query plugin in order to do just that. The idea is to select all document without a category and whose url matches a certain string and add the category you wish.
curl -XPOST 'localhost:9200/webproxylog/_update_by_query' -H "Content-Type: application/json" -d '
{
"query": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"term": {
"url": "stackoverflow.com"
}
},
{
"missing": {
"field": "category"
}
}
]
}
}
}
},
"script" : "ctx._source.category = \"10\";"
}'
After running this, all your documents with url: stackoverflow.com that don't have a category, will get category: 10. You can run the same query again later to fix new stackoverflow.com documents that have been indexed in the meantime.
Also make sure to enable scripting in elasticsearch.yml and restart ES:
script.inline: on
script.indexed: on
In the script, you're free to add as many fields as you want, e.g.
...
"script" : "ctx._source.category1 = \"10\"; ctx._source.category2 = \"20\";"
UPDATE
ES 2.3 now features the update by query functionality. You can still use the above query exactly as is and it will work (except that filtered and missing are deprecated, but still working ;).
That all sounds great but just to add to #Val answer, Update By Query is available form ElasticSearch 2.x but not for earlier versions. In our case we're using 1.4 for legacy reasons and there is no chance of upgrading in forseeable future so another solution is using the Update by query plugin provided here: https://github.com/yakaz/elasticsearch-action-updatebyquery