ElasticSearch function_score query with filters - elasticsearch

Trying to create a search that will bring back results of location that are about 500m away from certain geo point.
I need to filter results from this search based on if location is empty in the source, or not.
I tried things like this:
"filtered" : {
"filter": {
"missing" : { "field" : "location" }
}
}
This is the search JSON I got:
{"query": {
"function_score": {
"query": {
"bool": {
"must": [
{
"match": {"fieldA": "value"}
},
{
"match": { "fieldB": "value"}
}
]
}
},
"functions": [{
"gauss": {
"location": {
"origin": "'.$lat.','.$lon.'",
"scale": "500m",
"offset": "0km",
"decay": 0.33
}
}
}]
}
}
}
I tried putting the filter in different places in the query but it didn't work for me, so I know I'm doing something fundamentally wrong with how the query is structured. In the future I want to add more scoring logic and other filters, but can't find a good example of such queries.
What I should do to make it work?

You can use filtered query with geo_distance to filter out the results first. You can also use "weight" in function_score to explicit boost the distance over the "match" queries:
{
"query": {
"function_score": {
"query": {
"filtered": {
"query": {
"bool": {
"must": [
{
"match": {
"fieldA": "value"
}
},
{
"match": {
"fieldB": "value"
}
}
]
}
},
"filter": {
"geo_distance" : {
"distance" : "500m",
"location" : "'.$lat.','.$lon.'"
}
}
}
},
"functions": [
{
"gauss": {
"location": {
"origin": "'.$lat.','.$lon.'",
"scale": "500m",
"offset": "0km",
"decay": 0.33
}
},
"weight": "3"
}
]
}
}
}

Related

how to add filters to elastic query when using function_score?

Here is my current elastic query:
{
"from": 0,
"size": 10,
"query": {
"function_score": {
"query": {
"bool": {
"must": [{
"multi_match": {
"query": "ocean",
"fields": [],
"fuzziness": "AUTO"
}}],
"must_not": [{
"exists": {
"field": "parentId"
}
}]
}
},
"functions" : [
{
"gauss": {
"createTime": {
"origin": "2020-07-09T23:50:00",
"scale": "365d",
"decay": 0.3
}
}
}
]
}
}
}
How do I properly add filters to this? I think maybe the fact that I'm using function_score makes this different? I would like to add a hard filter, for example, only show me results with uploadUser: 'Mr. Bean' ... but still keep the scoring in place for the results that pass this filter.
I tried using filter in various places, also using must but I either get no results or all the results.
I'm using Elastic Search 7. Thanks for your help
You can try this below search query:
Refer this ES official documentation to know more about Function score query
{
"from": 0,
"size": 10,
"query": {
"function_score": {
"query": {
"bool": {
"filter": {
"term": {
"uploadUser": "Mr. Bean"
}
},
"must": [
{
"multi_match": {
"query": "ocean",
"fields": [
],
"fuzziness": "AUTO"
}
}
],
"must_not": [
{
"exists": {
"field": "parentId"
}
}
]
}
},
"functions": [
{
"gauss": {
"createTime": {
"origin": "2020-07-09T23:50:00",
"scale": "365d",
"decay": 0.3
}
}
}
]
}
}
}

function_score: treat missing field as perfect hit

What I need to do is boost documents by location (closer is better). locations is nested type.
Works fine except Elasticsearch does not return documents if locations is missing in document. If fields is missing, Elasticsearch should treat document as perfect hit. Any idea how to achieve this?
My query:
{
"sort": [
{
"_score": "desc"
}
],
"query": {
"function_score": {
"query": {
"bool": {
"must_not": [],
"should": [
{
"nested": {
"path": "locations",
"query": {
"function_score": {
"score_mode": "sum",
"functions": [
{
"gauss": {
"locations.coordinates": {
"origin": {
"lat": "50.1078852",
"lon": "15.0385376"
},
"scale": "100km",
"offset": "20km",
"decay": "0.5"
}
}
}
]
}
}
}
}
]
}
}
}
}
}
BTW: I'm using Elasticsearch 5.0
Add one more function into the functions section with high boost (like 1000) for missing locations, something like this:
{
"query": {
"bool": {
"must_not": {
"exists": {
"field": "locations"
}
}
},
},
"weight": 1000
}
So records with missing locations will come first because of high weight.
Syntax can differ a little. More information on queries here:
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-exists-query.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html

Elastic search how can I query either multi match or functions

I have three following parameters that I will pass to run the query, which are;
query - Either a place name, description or empty,
lat - Either latitude of a place or empty,
lon - Either longitude of a place or empty
Based on above parameters, I get to query list of items based on query scores, then calculate the distance between result and lat, lon.
Now, I have the following script to get the items based on query and distance;
{
"query": {
"function_score": {
"query": {
"bool": {
"must": [{
"multi_match" : {
"query": "Lippo",
"fields": [ "name^6", "city^5", "country^4", "position^3", "address_line^2", "description"]
}
}]
}
},
"functions": [
{
"gauss": {
"position": {
"origin": "-6.184652, 106.7518749",
"offset": "2km",
"scale": "10km",
"decay": 0.33
}
}
}
]
}
}
}
But the thing is, if query is empty, there will be no result at all. What I want is, the result is based on either query or distance.
Is there anyway to achieve this? Any suggestion is appreciated.
setting the zero_terms_query option of multi-match to all should allow you to get the results when query is empty.
Example :
{
"query": {
"function_score": {
"query": {
"bool": {
"must": [{
"multi_match" : {
"query": "Lippo",
"fields": [ "name^6", "city^5", "country^4", "position^3", "address_line^2", "description"],
"zero_terms_query" : "all"
}
}]
}
},
"functions": [
{
"gauss": {
"position": {
"origin": "-6.184652, 106.7518749",
"offset": "2km",
"scale": "10km",
"decay": 0.33
}
}
}
]
}
}
}

Elasticsearch No query registered for [exists]]

After running the below query, i am getting the exception as No query registered for [exists]. Please help me.
{
"query": {
"function_score": {
"query": {
"bool": {
"must": {
"match": {
"_all": {
"query": "cardio new york"
}
}
}
}
},
"functions": [
{
"gauss": {
"geo_location": {
"origin": {
"lat": 40.7127,
"lon": -74.0059
},
"scale": "100km",
"offset": "0km",
"decay": 0.9
}
}
},
{
"gauss": {
"startdate": {
"origin": "now",
"scale": "30d",
"offset": "30d"
}
},
"weight": 0.5
}
],
"filter": {
"query": {
"bool": {
"must": {
"match": {
"_all": {
"query": "cardio new york"
}
}
},
"should": {
"exists": {
"fields": [
"venue",
"geo_location"
]
}
}
}
}
}
}
}
}
I am trying to filter the search results after the function_score with combining bool match query.
exists is not a query, it's filter you cannot use it in a bool query, instead I would use bool filter and wrap only match into query filter like this:
"filter": {
"bool": {
"must": [{
"query": {
"match": {
"_all": {
"query": "cardio new york"
}
}
}
}, {
"exists": {
"fields": [
"venue",
"geo_location"
]
}
}]
}
}
This depends on your version, before version 2.0 you would use the Exists filter, as demonstrated in imotov's answer.
With 2.0 and after exists filter has been replaced by exists query documentation for current version
thus the newer exists query would look like the following:
{
"index": "foobars",
"type": "foo",
"body": {
"query": {
"bool": {
"must":
{"exists" : { "field" : "bar" }}
}
},
"from": 0,
"size": 20
}
}

Elasticsearch: query produced is invalid

I'm getting a little frustrated with elasticsearch, after having read the documents, but can't seem to get beyond a 'The query produced is invalid" response. What I am trying to do is use elasticsearch to find imperfect duplicates in geospatial information and a rather large dataset. I want to match on name (boosted) and address, filter results a small geographic box and then reduce the relevance score of matches that are located further from my reference point. Can someone please help? I think I understnad the individual elements of a query, my main problem is putting these together in a way that products something valid.
$query = new \Elastica\Query\Builder('{
"function_score": {
"functions": [
{
"gauss": {
"location": {
"origin": "'.$latitude.', '.$longitude.'",
"scale": "2km"
}
}
}
],
"query": {
"filtered": {
"query": {
"bool": {
"should": [
{
"match": {
"name": {
"query": "'.$name.'",
"boost": 4
}
}
},
{
"match": {
"address": {
"query": "'.$address.'",
"boost": 1
}
}
}
]
}
},
"filter": {
"geo_distance": {
"distance": "2km",
"location": {
"lat": "'.$latitude.'",
"lon": "'.$longitude.'"
}
}
}
}
}
}')
You should surround the whole query by a "query" clause:
$query = new \Elastica\Query\Builder('{
"query": {
"function_score": {
"functions": [
{
"gauss": {
"location": {
"origin": "'.$latitude.', '.$longitude.'",
"scale": "2km"
}
}
}
],
"query": {
"filtered": {
"query": {
"bool": {
"should": [
{
"match": {
"name": {
"query": "'.$name.'",
"boost": 4
}
}
},
{
"match": {
"address": {
"query": "'.$address.'",
"boost": 1
}
}
}
]
}
},
"filter": {
"geo_distance": {
"distance": "2km",
"location": {
"lat": "'.$latitude.'",
"lon": "'.$longitude.'"
}
}
}
}
}
}
}')
To have more feedback when making queries, a good habit is to print the Elasticsearch's response to the query.
Try the query in your SENSE:
POST test/_search
{
"function_score": {
"functions": [
{
"gauss": {
"location": {
"origin": "11, 12",
"scale": "2km"
}
}
}
],
"query": {
"filtered": {
"query": {
"bool": {
"should": [
{
"match": {
"name": {
"query": "name",
"boost": 4
}
}
},
{
"match": {
"address": {
"query": "address",
"boost": 1
}
}
}
]
}
},
"filter": {
"geo_distance": {
"distance": "2km",
"location": {
"lat": "'.$latitude.'",
"lon": "'.$longitude.'"
}
}
}
}
}
}
}
You'll get the response:
SearchParseException[[test][4]: from[-1],size[-1]: Parse Failure [No parser for element [function_score]]]; }]
Telling that Elasticsearch doesn't recognize "function_score". By reading in detail the function_score page on the elasticsearch wiki and looking at the example, you'll then find what's missing: the surrounding "query" clause.
PS: Elastica also provides an alternative syntax for creating the query body with the ElasticaQueryBuilder, which spare you from writing the json.

Resources