I am trying to define a watcher in kibana to monitor our logs and send a notification with message detail to our slack channel when an error happens.
Problem is that i can't find a way to get the message detail. I have printed the whole {{ ctx }} dict, and found nothing useful there.
You can use your query and hit elasticsearch and check the response for the message details. Below example is for multi-bucket aggregation query similarly you can query elasticsearch with you set of query and check the response.
curl -H "Content-Type: application/json" -X POST http://localhost:9002/filebeat-2019.04.04/_search?pretty=true -d '{"size": 0,"aggs": {"messages": {"filters": {"filters": {"errors": {"match": {"message": "CREDENTIALS"}},"warnings": {"match": {"message": "SUCCESS"}}}}}}}'
Response:
{
"took": 9,
"timed_out": false,
"_shards": ...,
"hits": ...,
"aggregations": {
"messages": {
"buckets": {
"errors": {
"doc_count": 1
},
"warnings": {
"doc_count": 2
}
}
}
}
}
You should be able to find the message element from the query and condition used for the watcher.
For example, if your condition was something like this:
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gt": 0
}
}
}
For this case, the message can be found at this location:
ctx.payload.hits.hits.0._source.message
Related
Just now a strange thing happened. I used ran a search request on an elasticsearch cluster.
curl http://<cluster>/<index>/_search -H 'Content-Type: application/json' -d' { "query": { "match": { "<field>": "<value>" } } } '
The response was
{ "hits": { "hits": [{ "_id": "<the ID>", ... }], ... }, ... }
Then I tried to query this doc.
curl http://<cluster>/<index>/_doc/<the ID>
But it responded
{"_index":"<index>","_type":"_doc","_id":"<the ID>","found":false}
I tried to delete this record, but it was also not found.
{"_index":"<index>","_type":"_doc","_id":"<the ID>","_version":1,"result":"not_found","_shards":{ ... }, ... }
I finally used delete by query.
What could possibily cause this issue?
Elasticsearch version: 7.10.2
It usually means that the record was indexed with a routing value, which means you need to provide that routing value to retrieve it, update it and/or delete it.
You can try different routing values, up to the number of primary shards that you have, i.e. if you have 5 primary shards, one of the following must return your document
curl http://<cluster>/<index>/_doc/<the ID>?routing=0
curl http://<cluster>/<index>/_doc/<the ID>?routing=1
curl http://<cluster>/<index>/_doc/<the ID>?routing=2
curl http://<cluster>/<index>/_doc/<the ID>?routing=3
curl http://<cluster>/<index>/_doc/<the ID>?routing=4
You can find the routing value used for your document in the search response in the _routing field value.
{ "hits": { "hits": [{ "_id": "<the ID>", "_routing": "<the ROUTING>", ... }], ... }, ... }
Need to find unique string values that are in list field.
The question is similar to ElasticSearch - Return Unique Values
but now field values are lists
Records:
PUT items/1
{ "tags" : ["a", "b"] }
PUT items/2
{ "tags" : ["b", "c"] }
PUT items/3
{ "tags" : ["a" "d"] }
Query:
GET items/_search
{ ... }
# => Expected Response
["a", "b", "c", "d"]
Is there way to make such search?
Good news! We can use the exact same aggregation as the one used in the SO post you linked to in the description. In fact, if we were submitting a list of numeric values, our work would be done already! However the main difference between this question and the question you referenced is that you are using a "string" type.
It is useful to know that in more recent versions of elasticsearch, there are two ways to represent "strings" in elasticsearch and that type is actually not referred to as a string any more. Using the keyword type will treat the entire text as a single token, while using the text type will apply an analyzer to break the text up into many different tokens and build an index with those tokens.
For example, the string "Foxes are brown" can be represented as "foxes are brown" or ["foxes", "are", "brown"] in the index. In your case, tags should be treated as a keyword so we'll need to tell elasticsearch that that field is a keyword and not text which is the default.
NOTE: Using the keyword type whenever possible will alleviate the issue of needing to allow elasticsearch to set fielddata to true, which uses up a lot of memory in your cluster if this aggregation is used much. Tags and ordinal data are good candidates for the keyword type.
Anyways, let's get to the real stuff eh?
First, you're going to want to set the mapping for tags in the items as a keyword type.
curl --request PUT \
--url http://localhost:9200/items \
--header 'content-type: application/json' \
--data '{
"mappings": {
"item": {
"properties": {
"tags" : { "type": "keyword" }
}
}
}
}
'
Then you're going to run the aggregation similar to the aggregation in the post you referenced.
curl --request POST \
--url http://localhost:9200/items/item/_search \
--header 'content-type: application/json' \
--data '{
"size": 0,
"aggregations": {
"tags_term_agg": {
"terms": {
"field": "tags"
}
}
}
}'
Your response should looks something like this.
{
"took": 24,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 3,
"max_score": 0.0,
"hits": []
},
"aggregations": {
"tags_term_agg": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "a",
"doc_count": 2
},
{
"key": "b",
"doc_count": 2
},
{
"key": "c",
"doc_count": 1
},
{
"key": "d",
"doc_count": 1
}
]
}
}
}
I want to check if a document with specific field value does exist or not in elasticsearch.
I have gone through internet but only found how to check if a field exists or not.
My Index/Type is
/twitter/user
username is one field in document.
I want to check if username="xyz" exists or not in this Type.
You can query with size 0. total value will give you an idea that doc exists or not.
GET /twitter/user/_search
{"size": 0,
"query": {"match": {
"username": "xyz"
}}}
Edited --
_count api can be used as well.
GET /twitter/user/_count
{ "query": {"match": {
"username": "xyz"
}}}
From the documentation:
If all you want to do is to check whether a document exists—you’re not interested in the content at all—then use the HEAD method instead of the GET method. HEAD requests don’t return a body, just HTTP headers:
curl -i -XHEAD http://localhost:9200/twitter/user/userid
Elasticsearch will return a 200 OK status code if the document exists ... and a 404 Not Found if it doesn’t exist
Note: userid is the value of the _id field.
simply search the document, if it exists it will return result otherwise not
http://127.0.0.1:9200/twitter/user/_search?q=username:xyz
and the exact what are you looking for is
http://127.0.0.1:9200/twitter/user/_search/exists?q=username:xyz
it will return exists true or false
{
"exists": false
}
You can use term query with size 0. See below query for reference:
POST twitter/user/_search
{
"query": {
"term": {
"username": "xyz"
}
},
"size" : 0
}
Response:
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 0,
"hits": []
}
}
You will get total document count in hits.total and then you can check for count > 0
In some cases, I don't need all of the fields in response json.
For example,
// request json
{
"_source": "false",
"aggs": { ... },
"query": { ... }
}
// response json
{
"took": 123,
"timed_out": false,
"_shards": { ... },
"hits": {
"total": 123,
"max_score": 123,
"hits": [
{
"_index": "foo",
"_type": "bar",
"_id": "123",
"_score": 123
}
],
...
},
"aggregations": {
"foo": {
"buckets": [
{
"key": 123,
"doc_count": 123
},
...
]
}
}
}
Actually I don't need the _index/_type every time. When I do aggregations, I don't need hits block.
"_source" : false or "_source": { "exclude": [ "foobar" ] } can help ignore/exclude the _source fields in hits block.
But can I change the structure of ES response json in a more common way? Thanks.
I recently needed to "slim down" the Elasticsearch response as it was well over 1MB in json and I started using the filter_path request variable.
This allows to include or exclude specific fields and can have different types of wildcards. Do read the docs in the link above as there is quite some info there.
eg.
_search?filter_path=aggregations.**.hits._source,aggregations.**.key,aggregations.**.doc_count
This reduced (in my case) the response size by half without significantly increasing the search duration, so well worth the effort..
In the hits section, you will always jave _index, _type and _id fields. If you want to retrieve only some specific fields in your search results, you can use fields parameter in the root object :
{
"query": { ... },
"aggs": { ... },
"fields":["fieldName1","fieldName2", etc...]
}
When doing aggregations, you can use the search_type (documentation) parameter with count value like this :
GET index/type/_search?search_type=count
It won't return any document but only the result count, and your aggregations will be computed in the exact same way.
I am trying to the example from the elastic search site with my own parameters, but it is not working.
Query:
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"range": {
"activity_date": {
"from": "2013-11-01",
"to": "2014-11-01"
}
}
}
}
},
"aggs": {
"net_ordered_units": {
"sum": {
"field": "net_ordered_units"
}
}
}
}
Error I get:
{
"error": "SearchPhaseExecutionException[Failed to execute phase [query_fetch], all shards failed; shardFailures {[YoGKlejVTC6jhg_OgPWXyTg][test][0]: SearchParseException[[test][0]: query[ConstantScore(cache(activity_date:[1383264000000 TO 1414886399999]))],from[-1],size[-1]: Parse Failure [Failed to parse source [{\"query\": {\"filtered\":{\"query\":{\"match_all\":{}},\"filter\":{\"range\":{\"activity_date\":{\"from\":\"2013-11-01\",\"to\":\"2014-11-01\"}}}}},\"aggs\":{\"net_ordered_units\":{\"sum\": {\"field\":\"net_ordered_units\"}}}}]]]; nested: SearchParseException[[test][0]: query[ConstantScore(cache(activity_date:[1383264000000 TO 1414886399999]))],from[-1],size[-1]: Parse Failure [No parser for element [aggs]]]; }]",
"status": 400
}
What is shard failure here? And it says no parser for aggs, what should I do here?
Basically, I need to perform operations like sum and then find the max out of it.
How should I modify the above code to get that?
I think your plugin (which you use to perform the CURL based elastic-search queries) is not able to parse the "aggs" tag. I use the Marvel Sense plugin (http://www.elasticsearch.org/guide/en/marvel/current/) specifically for ES queries and your query works fine ! I did a test on Postman ( a RESTful Chrome Plugin) and guess what, nothing wrong with your query... So try switching your plugin and see if that helps.
Updated:
To answer the second part of your question,
curl -s -XPOST your_ES_server/ES_index/url_to_query -d
'{"query":
{"bool":
{
"must": [{
"wildcard" : { "item_id" : "*" }
}]
}
},
"facets" : {
"facet_result":
{"terms":{
"fields":["item_count"]
}}
}
Gotcha, Actually the above query doesn't fetch you the maximum count of a specific field key but lists you all the field keys sorted by their count in descending order(by default). So naturally the top most term should be what you are looking for. The response to the above query looks as follows.
"facets": {
"facet_result": {
"_type": "terms",
"missing": 0,
"total": 35,
"other": 0,
"terms": [
{
"term": 0,
"count": 34
},
{
"term": 2,
"count": 1
}
]
}
}
This might not be a clean solution but can help you retrieve the max(sum) of a key. For more info on ordering, refer http://www.elasticsearch.org/guide/en/elasticsearch/reference/0.90/search-facets-terms-facet.html#_ordering