I am trying to query elasticsearch for all logs that have a #timestamp within a certain range. This is being done by a $.get() request using jQuery. My attempts so far have resulted in queries not within the time range. I am listing the logs on my own HTML page and although I have tried different query messages I always get one of two responses, one with times starting at 4:25 and the other with times starting at 6:39. I don't know if this is useless information but I thought it could be useful in understanding the nature of the problem.
So this is the message I have written myself.
{
"query" : {
"filtered": {
"query" : { "match_all" : {}},
"filter": {
"range" : {
"#timestamp": {
"from" : 1393580102248,
"to" : 1393583702248
}
}
}
}
}
}
We are also using Kibana (working) and I have tried copying and pasting the same exact query Kibana uses, with the SAME results as my query above.
{
"query": {
"filtered": {
"query": {
"bool": {
"should": [
{
"query_string": {
"query": "*"
}
}
]
}
},
"filter": {
"bool": {
"must": [
{
"match_all": {}
},
{
"range": {
"#timestamp": {
"from": 1393580102248,
"to": 1393583702248
}
}
},
{
"bool": {
"must": [
{
"match_all": {}
}
]
}
}
]
}
}
}
},
"highlight": {
"fields": {},
"fragment_size": 2147483647,
"pre_tags": [
"#start-highlight#"
],
"post_tags": [
"#end-highlight#"
]
},
"size": 500,
"sort": [
{
"#timestamp": {
"order": "desc"
}
}
]
}
What am I doing wrong? I can change the time range as well and I still get the same results.
Related
I have a function_score query and i want to apply top_hit aggregation on the output of my function_score query, as such i am using function_score query to filter parents based child properties, then i want to run some queries over those parents via the aggregation and then sort them accordingly.
Query:
POST test/_search?size=0
{
"query": {
"function_score": {
"query": {
"match_all" : {}
},
"functions": [
{
"filter": {
"bool": {
"must": [
{
"has_child": {
"type": "track",
"query": {
"exists": {
"field": "url"
}
}
}
}
]
}
},
"weight": 5
}
]
}
},
"aggs": {
"top_tags": {
"filter": {
"bool": {
"must": [
{
"exists": {
"field": "common_info.CD.Handle"
}
},
{
"match_phrase" : {
"common_info.CD.device" : "D/C"
}
}
]
}
},
"aggs": {
"test_agg_on_doc": {
"top_hits": {
"sort" : [
],
"_source": {
"includes": [
"common_info.CD.Handle"
]
},
"size": 1
}
}
}
}
}
}
When i run this query my function_score query is not getting considered at all as such and "aggs" are working on the total number of docs, but i want it to run on the docs that are filtered using function_score query. Any help would be highly appreciated. thanks
I've been facing some issues with multi field elasticsearch query. I am trying to query all the documents which matches the field called func_name to two hard coded strings, even though my index has documents with both these function names, but the query result is always fetching only one func_name. So far I have tried following queries.
1) Following returns only one function match, even though the documents have another function as well
GET /_search
{
"query": {
"multi_match": {
"query": "FEM_DS_GetTunerStatusInfo MDM_TunerStatusPrint",
"operator": "OR",
"fields": [
"func_name"
]
}
}
}
2) following intermittently gives me both the functions.
GET /_search
{
"query": {
"match": {
"func_name": {
"query": "MDM_TunerStatusPrint FEM_DS_GetTunerStatusInfo",
"operator": "or"
}
}
}
}
3) Following returns only one function match, even though the documents have another function as well
{
"query": {
"bool": {
"should": [
{ "match": { "func_name": "FEM_DS_GetTunerStatusInfo" }},
{ "match": { "func_name": "MDM_TunerStatusPrint" }}
]
}
}
}
Any help is much appreciated.
Thanks for your reply. Lets assume that I have following kind of documents in my elasticsearch. I want my search to return first two documents out of all as they matches my func_name.
{
"_index": "diag-178999",
"_source": {
"severity": "MIL",
"t_id": "03468500",
"p_id": "000007c6",
"func_name": "MDM_TunerStatusPrint",
"timestamp": "2017-06-01T02:04:51.000Z"
}
},
{
"_index": "diag-344563",
"_source": {
"t_id": "03468500",
"p_id": "000007c6",
"func_name": "FEM_DS_GetTunerStatusInfo",
"timestamp": "2017-07-20T02:04:51.000Z"
}
},
{
"_index": "diag-101010",
"_source": {
"severity": "MIL",
"t_id": "03468500",
"p_id": "000007c6",
"func_name": "some_func",
"timestamp": "2017-09-15T02:04:51.000Z"
}
The "two best ways" to request your ES is to filter by terms on a particular field or to aggregate your queries so that you can rename the field, apply multiple rules, and give a more understandable format to your response
See : https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html and the other doc page is here, very useful :
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html
In your case, you should do :
{
"from" : 0, "size" : 2,
"query": {
"filter": {
"bool": {
"must": {
"term": {
"func_name" : "FEM_DS_GetTunerStatusInfo OR MDM_TunerStatusPrint",
}
}
}
}
}
}
OR
"aggs": {
"aggregationName": {
"terms": {
"func_name" : "FEM_DS_GetTunerStatusInfo OR MDM_TunerStatusPrint"
}
}
}
}
The aggregation at the end is just here to show you how to do the same thing as your query filter. Let me know if it's working :)
Best regards
As I understand, you should use filtered query to match any document with one of the values of func_name mentioned above:
{
"query": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"terms": {
"func_name": [
"FEM_DS_GetTunerStatusInfo",
"MDM_TunerStatusPrint"
]
}
}
]
}
}
}
}
}
See:
Filtered Query, Temrs Query
UPDATE in ES 5.0:
{
"query": {
"bool": {
"must": [
{
"terms": {
"func_name": [
"FEM_DS_GetTunerStatusInfo",
"MDM_TunerStatusPrint"
]
}
}
]
}
}
}
See: this answer
I am using logstash to store logs in elasticsearch database. I want to get logs having a particular severitylabel and are between certain time stamps and matches to some specific message. The curl query I wrote is :
curl -XPOST 'localhost:9200/logstash-2015.06.19/_search/?pretty' -d '{
"query": {
"filtered": {
"query": {
"bool": {
"must": [
{
"match": {
"#message": "session"
}
}
]
}
},
"filter": {
"bool": {
"must": [
{
"range": {
"#timestamp": {
"gte": "2015-06-19T10:11:44.000Z",
"lte": "2015-06-19T11:11:44.000Z"
}
}
},
{
"term": {
"#app": "sparta"
}
},
{
"terms": {
"#severityLabel": [
"INFO",
"WARN",
"ERROR",
"FATAL",
"OFF"
]
}
}
]
}
}
}
} } '
It always shows zero documents, matched. I am using term filter as a sibling of terms filter, is that a problem?
My filters are grouped together into categories.
I would like to retrieve documents where a document can match any filter in a category, but if two (or more) categories are set, then the document must match any of the filters in ALL categories.
If written in pseudo-SQL it would be:
SELECT * FROM Documents WHERE (CategoryA = 'A') AND (CategoryB = 'B' OR CategoryB = 'C')
I've tried Nested filters like so:
{
"sort": [{
"orderDate": "desc"
}],
"size": 25,
"query": {
"match_all": {}
},
"filter": {
"and": [{
"nested": {
"path":"hits._source",
"filter": {
"or": [{
"term": {
"progress": "incomplete"
}
}, {
"term": {
"progress": "completed"
}
}]
}
}
}, {
"nested": {
"path":"hits._source",
"filter": {
"or": [{
"term": {
"paid": "yes"
}
}, {
"term": {
"paid": "no"
}
}]
}
}
}]
}
}
But evidently I don't quite understand the ES syntax. Is this on the right track or do I need to use another filter?
This should be it (translated from given pseudo-SQL)
{
"sort": [
{
"orderDate": "desc"
}
],
"size": 25,
"query":
{
"filtered":
{
"filter":
{
"and":
[
{ "term": { "CategoryA":"A" } },
{
"or":
[
{ "term": { "CategoryB":"B" } },
{ "term": { "CategoryB":"C" } }
]
}
]
}
}
}
}
I realize you're not mentioning facets but just for the sake of completeness:
You could also use a filter as the basis (like you did) instead of a filtered query (like I did). The resulting json is almost identical with the difference being:
a filtered query will filter both the main results as well as facets
a filter will only filter the main results NOT the facets.
Lastly, Nested filters (which you tried using) don't relate to 'nesting filters' like you seemed to believe, but related to filtering on nested-documents (parent-child)
Although I have not understand completely your structure this might be what you need.
You have to think tree-wise. You create a bool where you must (=and) fulfill the embedded bools. Each embedded checks if the field does not exist or else (using should here instead of must) the field must (terms here) be one of the values in the list.
Not sure if there is a better way, and do not know the performance.
{
"sort": [
{
"orderDate": "desc"
}
],
"size": 25,
"query": {
"query": { #
"match_all": {} # These three lines are not necessary
}, #
"filtered": {
"filter": {
"bool": {
"must": [
{
"bool": {
"should": [
{
"not": {
"exists": {
"field": "progress"
}
}
},
{
"terms": {
"progress": [
"incomplete",
"complete"
]
}
}
]
}
},
{
"bool": {
"should": [
{
"not": {
"exists": {
"field": "paid"
}
}
},
{
"terms": {
"paid": [
"yes",
"no"
]
}
}
]
}
}
]
}
}
}
}
}
I am trying to add the partial_fields directive to an elasticsearch query (generated from kibana's table widget).
Where exactly would I have to place this statement in the below ES query?
Already tried to add it right after the first "query" node which produces valid json but still doesn't exclude xmz_Data
"partial_fields": {
"partial1": {
"exclude": "xmz_Data"
}
},
ES Query
{
"query": {
"filtered": {
"query": {
"bool": {
"should": [
{
"query_string": {
"query": "*"
}
}
]
}
},
"filter": {
"bool": {
"must": [
{
"match_all": {}
},
{
"bool": {
"must": [
{
"match_all": {}
}
]
}
}
]
}
}
}
},
"highlight": {
"fields": {},
"fragment_size": 2147483647,
"pre_tags": [
"#start-highlight#"
],
"post_tags": [
"#end-highlight#"
]
},
"size": 250,
"sort": [
{
"timestamp": {
"order": "desc"
}
}
]
}
You can place the partial_fields directive anywhere in your query, I tested successfully with it both before and after the query node. However, your formatting for the excluded fields value is incorrect. Your exclude fields value needs to be an array. Try this instead...
"partial_fields": {
"partial1": {
"exclude": ["xmz_Data"]
}
},