Is there any way to get documents from elasticsearch in custom order .
the documents in elastic are indexed like this :
{
"productId":1
},
{
"productId":2
},
{
"productId":3
}
i need to get documents in order as term query order
i try this query but not work :
{
"from":0,
"size":20,
"query":{
"bool":{
"should":[
{
"term":{
"productId":{
"value":3,
"boost":1.0
}
}
},
{
"term":{
"productId":{
"value":1,
"boost":1.0
}
}
},
{
"term":{
"productId":{
"value":2,
"boost":1.0
}
}
}
],
"adjust_pure_negative":true,
"boost":1.0
}
},
"version":true
}
as you see term queries with productId:3 , productId:1 , productId:2
i expect result are documents sorted as term queries order :
{
"productId":3
},
{
"productId":1
},
{
"productId":2
}
Related
In the below query, I used multiple times match_phrase. how to reduce multiple match_phrase? because in production while querying to ES response is very slow.
GET /logs*/_search
{
"from":0,
"query":{
"bool":{
"filter":[
{
"range":{
"#timestamp":{
"gte":"2020-02-10T11:13:19.7684961Z",
"lte":"2020-02-11T11:13:19.7684961Z"
}
}
}
],
"must":[
{
"bool":{
"must_not":[
{
"match_phrase":{
"message":{
"query":"System32"
}
}
},
{
"match_phrase":{
"message":{
"query":"212.118.14.45"
}
}
},
{
"match_phrase":{
"message":{
"query":" stopped state."
}
}
},
{
"match_phrase":{
"message":{
"query":" running state"
}
}
},
{
"match_phrase":{
"message":{
"query":" Share Name: \\\\*\\DLO-EBackup"
}
}
}
.
.
.
etc.,
.
.
.
.
.
{
"match_phrase":{
"message":{
"query":"WFO15Installation"
}
}
},
{
"match_phrase":{
"message":{
"query":"Windows\\SysWOW64"
}
}
},
{
"match_phrase":{
"message":{
"query":"Bitvise"
}
}
}
]
}
}
]
}
},
"size":10,
"sort":[
{
"#timestamp":{
"order":"desc"
}
}
]
}
Thank You!
to begin with, you could move the must_not block inside the filter one to skip score calculation and leverage on some caching. Something like:
"query":{
"bool":{
"filter":[{
"range":{
"#timestamp":{
"gte":"2020-02-10T11:13:19.7684961Z",
"lte":"2020-02-11T11:13:19.7684961Z"
}
}
},
{
"bool": {
"must_not":[{
"match_phrase":{
"message":{
"query":"System32"
}
}
},
{
"match_phrase":{
"message":{
"query":"212.118.14.45"
}
}
},
...
]
}
}],
...
However, as someone already mentioned in the comments, you should optimise your data for searches before indexing your documents into Elasticsearch. A better solution than having so many filters in your query would be to process your data and applying those filters at ingestion time, for example by using the ingest APIs (see Elastic Documentation) or Logstash. E.g., you could evaluate the must_not conditions at index time and set the result into a boolean field (e.g., ignore) that you can add to all documents, so that you can use that field at query time with a query like this:
"query":{
"bool":{
"filter":[{
"range":{
"#timestamp":{
"gte":"2020-02-10T11:13:19.7684961Z",
"lte":"2020-02-11T11:13:19.7684961Z"
}
}
},
{
"match": {
"ignore": false
}
},
...
I am running following elasticsearch query from groovy script. There are thousands of records which meet this criteria, but I get only 10 records in return.
{
"query":{
"bool":{
"must":[
{
"match_all":{
}
},
{
"range":{
"#Timestamp":{
"gte":1417511269270,
"lte":1575277669270,
"format":"epoch_millis"
}
}
},
{
"match_phrase":{
"field1.keyword":{
"query":"value1"
}
}
},
{
"match_phrase":{
"field2.keyword":{
"query":"value2"
}
}
},
{
"range":{
"#Timestamp":{
"gte":"2001-03-01",
"lt":"2019-10-30"
}
}
}
],
"filter":[
],
"should":[
],
"must_not":[
]
}
}
}
What am I missing in my query?
You are missing a size parameter, which means it defaults to 10 results.
e.g. add this to your query object:
"size": 100
I am running into some serious Problems with a custom Search. All i want is a Wildcard Search in three Fields and the Result should to filtered by another field. In Elastica it results in this Query:
{
"bool":{
"should":[
{
"wildcard":{
"ean":"*180g*"
}
},
{
"wildcard":{
"titel":"*180g*"
}
},
{
"wildcard":{
"interpret":"*180g*"
}
}
],
"filter":[
{
"term":{
"genre":{
"value":"Rock",
"boost":1
}
}
}
]
}
}
Actually i can't find an error, but Elasticsearch does not give me Filtered Results. What happens? Elasticsearch returns ALL Items with the Filtered Term, either if the Boolean Shoulds are True or False. When i add the Filter as "Must" i am getting the same results? What is wrong here !?
You need to add "minimum_should_match": 1 in your bool query.
{
"bool":{
"minimum_should_match": 1,
"should":[
{
"wildcard":{
"ean":"*180g*"
}
},
{
"wildcard":{
"titel":"*180g*"
}
},
{
"wildcard":{
"interpret":"*180g*"
}
}
],
"filter":[
{
"term":{
"genre":{
"value":"Rock",
"boost":1
}
}
}
]
}
}
I've got the following query
{
"from":0,
"size":50000,
"_source":[
"T121",
"timestamp"
],
"sort":{
"timestamp":{
"order":"asc"
}
},
"query":{
"bool":{
"must":{
"range":{
"timestamp":{
"gte":"2017-01-17 11:44:41.347",
"lte":"2017-02-18 11:44:47.878"
}
}
},
"must":{
"exists":{
"field":"T121"
}
}
}
}
}
http://172.22.23.169:9200/index1,index2,Index3/_search?pretty
With this URL i want to query over a number of indices in Elasticsearch and only return those documents where a specific field exists.
Is it possible to put in a list of fields in the "exists" clause where i define
if "field1" OR "field2" OR "fiedl3" are existing in one of the documents return it, otherwise don't, or do i have to script such a case?
To search across all indices use > http://172.22.23.169:9200/_search?pretty
To search across selected indices add following filter to "bool" filter
"must": {
"terms": {
"_index": [
"index1",
"index2"
]
}
}
For OR'ing multiple "exists", you can use should clause with multiple exists and specify "minimum_should_match" to control searched records.
{
"from":0,
"size":50000,
"_source":[
"T121",
"timestamp"
],
"sort":{
"timestamp":{
"order":"asc"
}
},
"query":{
"bool":{
"must":{
"range":{
"timestamp":{
"gte":"2017-01-17 11:44:41.347",
"lte":"2017-02-18 11:44:47.878"
}
}
},
"should":[
{
"exists":{
"field":"field1"
}
},
{
"exists":{
"field":"field2"
}
},
{
"exists":{
"field":"field3"
}
}
]
}
}
}
I would like to make a query with two subqueries, each of them has it's own scoring based on function_score with script. For example, this subquery:
{
"query":{
"function_score":{
"query":{
"bool":{
"filter":[
{
"term":{
"rooms_count":3
}
},
{
"term":{
"addresses":"d76255c8-3173-4db5-a39b-badd3ebdf851"
}
},
{
"exists":{
"field":"zhk_id"
}
}
]
}
},
"script_score":{
"script":"1 * doc['price'].value/100000"
},
"boost_mode":"replace"
}
}
}
works fine, and it's score is based on price (about 190 points). But if I try to combine two subqueries in dis_max query, function_score is not working and I get scores about 1 point.
Explanation for each subquery looks like this
"value": 100.9416, "description": "script score function, computed with script:"[script: 1 * doc['price'].value/100000, type: inline, lang: null, params: {}]" and parameters:
{}",
for dis_max query like
"value": 1, "description": "ConstantScore(function score (#rooms_count: #addresses:d76255c8-3173-4db5-a39b-badd3ebdf851 #ConstantScore(fieldnames:zhk_id),function=script[script: 1 * doc['price'].value/100000, type: inline, lang: null, params: {}])), product of:",`
Can anybody tell me, how to combine function_score queries properly?
My full dis_max query on pastebin
Thanks to Daniel Mitterdorfer from https://discuss.elastic.co/t/combine-two-function-score-queries-in-dis-max/70666.
correct query is
{
"query":{
"dis_max":{
"queries":[
{
"function_score":{
"query":{
"bool":{
"filter":[
{
"term":{
"rooms_count":3
}
},
{
"term":{
"addresses":"d76255c8-3173-4db5-a39b-badd3ebdf851"
}
},
{
"missing":{
"field":"zhk_id"
}
}
]
}
},
"script_score":{
"script":"1 * doc['price'].value/100000"
},
"boost_mode":"replace"
}
},
{
"function_score":{
"query":{
"bool":{
"filter":[
{
"term":{
"rooms_count":3
}
},
{
"term":{
"addresses":"d76255c8-3173-4db5-a39b-badd3ebdf851"
}
},
{
"exists":{
"field":"zhk_id"
}
}
]
}
},
"script_score":{
"script":"1 * doc['price'].value/100000"
},
"boost_mode":"replace"
}
}
]
}
}
}