With Elasticsearch function score query with decay against a geo-point, is it possible to set a target distance? - elasticsearch

It doesn't seem possible based on https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html , but I'd like confirmation.
In plain English, I'm asking to score the results (with geo-point locations) by how close they are to 500km from some latitude, longitude origin.
It's confusing because there is a parameter called "offset" but according to the documentation it doesn't seem to be an offset from origin (eg. distance) but instead seems to mean "threshold" instead.

I see a few ways to accomplish this:
A. One way would be to simply sort by distance in reverse order from the origin. You'd use a geo_distance query and then sort by distance. In the following query, the most distant documents will come up first, i.e. the sort value is the distance from the origin and we're sorting in decreasing order.
{
"query": {
"filtered": {
"filter": {
"geo_distance": {
"from" : "100km",
"to" : "200km",
"location": {
"lat": 10,
"lon": 20
}
}
}
}
},
"sort": [
{
"_geo_distance": {
"location": {
"lat": 10,
"lon": 20
},
"order": "desc",
"unit": "km",
"distance_type": "plane"
}
}
]
}
B. The second way involves using a geo_distance_range query in order to define a "ring" around the origin. The width of that ring could somehow symbolize the offset + scale you'd use in a gauss function (although there would be no decay). Here we define a ring that is 10km wide at 500km distance from the origin point and sort the documents by distance in that ring.
{
"query": {
"filtered": {
"filter": {
"geo_distance_range": {
"from": "495km",
"to": "505km",
"location": {
"lat": 10,
"lon": 20
}
}
}
}
},
"sort": [
{
"_geo_distance": {
"location": {
"lat": 10,
"lon": 20
},
"order": "desc",
"unit": "km",
"distance_type": "plane"
}
}
]
}
C. The last way is a bit more involved. We're basically after an "inverse gauss" shape, basically this figure (33), but upside-down, or this one which better represents the donut shape we're after. We can combine solution B above with a gauss function that would only score within that ring. In the query below, we're basically saying that we're only interested in the locations around 500km from the origin and we let a gauss function kick in only for those documents. It's not perfect, though, but might be close enough to what you need.
{
"query": {
"filtered": {
"filter": {
"geo_distance_range": {
"from": "495km",
"to": "505km",
"location": {
"lat": 10,
"lon": 20
}
}
},
"query": {
"function_score": {
"functions": [
{
"gauss": {
"location": {
"origin": {
"lat": 10,
"lon": 20
},
"offset": "500km",
"scale": "5km"
}
}
}
]
}
}
}
},
"sort": {
"_score": "desc"
}
}

Related

ElasticSearch - Filtering a result and manipulating the documents

I have the following query - which works fine (this might not be the actual query):
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "location",
"query": {
"geo_distance": {
"distance": "16090km",
"distance_type": "arc",
"location.point": {
"lat": "51.794177",
"lon": "-0.063055"
}
}
}
}
},
{
"geo_distance": {
"distance": "16090km",
"distance_type": "arc",
"location.point": {
"lat": "51.794177",
"lon": "-0.063055"
}
}
}
]
}
}
}
Although I want to do the following (as part of the query but not affecting the existing query):
Find all documents that have field_name = 1
On all documents that have field_name = 1 run ordering by geo_distance
Remove duplicates that have field_name = 1 and the same value under field_name_2 = 2 and leave the closest item in the documents result, but remove the rest
Update (further explanation):
Aggregations can't be used as we want to manipulate the documents in the result.
Whilst also maintaining the order within the documents; meaning:
If I have 20 documents, sorted by a field; and I have 5 of which have field_name = 1, I would like to sort the 5 by distance, and eliminate 4 of them; whilst still maintaining the first sort. (possibly doing the geodistance sort and elimination before the actual query?)
Not too sure how to do this, any help is appreciated - I'm currently using ElasticSearch DSL DRF - but I can easily convert the query to ElasticSearch DSL.
Example documents (before manipulation):
[{
"field_name": 1,
"field_name_2": 2,
"location": ....
},
{
"field_name": 1,
"field_name_2": 2,
"location": ....
},
{
"field_name": 55,
"field_name_5": 22,
"location": ....
}]
Output (Desired):
[{
"field_name": 1,
"field_name_2": 2,
"location": .... <- closest
},
{
"field_name": 55,
"field_name_5": 22,
"location": ....
}]
One way to achieve what you want is to keep the query part as you have it now (so you still get the hits you need) and add an aggregation part in order to get the closest document with an additional condition on filed_name. The aggregation part would be made of:
a filter aggregation to only consider the documents with field_name = 1
a geo_distance aggregation with a very small distance
a top_hits aggregation to return the document with the closest distance
The aggregation part would look like this:
{
"query": {
...same as you have now...
},
"aggs": {
"field_name": {
"filter": {
"term": {
"field_name": 1 <--- only select desired documents
}
},
"aggs": {
"geo_distance": {
"field": "location.point",
"unit": "km",
"distance_type": "arc",
"origin": {
"lat": "51.794177",
"lon": "-0.063055"
},
"ranges": [
{
"to": 1 <---- single bucket for docs < 1km (change as needed)
}
]
},
"aggs": {
"closest": {
"top_hits": {
"size": 1, <---- closest document
"sort": [
{
"_geo_distance": {
"location.point": {
"lat": "51.794177",
"lon": "-0.063055"
},
"order": "asc",
"unit": "km",
"mode": "min",
"distance_type": "arc",
"ignore_unmapped": true
}
}
]
}
}
}
}
}
}
}
This can be done using Field Collapsing - which is the equivalent of grouping. - Below is an example of how this can be achieved:
{"collapse": {"field": "vin",
"inner_hits": {
"name": "closest_dealer",
"size": 1,
"sort": [
{
"_geo_distance": {
"location.point": {
"lat": "latitude",
"lon": "longitude"
},
"order": "desc",
"unit": "km",
"distance_type": "arc",
"nested_path": "location"
}
}
]
}
}
}
The collapsing is done on the field vin - and the inner_hits is used to sort the grouped items and get the closest one. (size = 1)

Elastic Search Geo Spatial search implementation

I am trying to understand how elastic search supports Geo Spatial search internally.
For the basic search, it uses the inverted index; but how does it combine with the additional search criteria like searching for a particular text within a certain radius.
I would like to understand the internals of how the index would be stored and queried to support these queries
Text & geo queries are executed separately of one another. Let's take a concrete example:
PUT restaurants
{
"mappings": {
"properties": {
"location": {
"type": "geo_point"
},
"menu": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
}
}
}
}
POST restaurants/_doc
{
"name": "rest1",
"location": {
"lat": 40.739812,
"lon": -74.006201
},
"menu": [
"european",
"french",
"pizza"
]
}
POST restaurants/_doc
{
"name": "rest2",
"location": {
"lat": 40.7403963,
"lon": -73.9950026
},
"menu": [
"pizza",
"kebab"
]
}
You'd then match a text field and apply a geo_distance filter:
GET restaurants/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"menu": "pizza"
}
},
{
"geo_distance": {
"distance": "0.5mi",
"location": {
"lat": 40.7388,
"lon": -73.9982
}
}
},
{
"function_score": {
"query": {
"match_all": {}
},
"boost_mode": "avg",
"functions": [
{
"gauss": {
"location": {
"origin": {
"lat": 40.7388,
"lon": -73.9982
},
"scale": "0.5mi"
}
}
}
]
}
}
]
}
}
}
Since the geo_distance query only assigns a boolean value (--> score=1; only checking if the location is within a given radius), you may want to apply a gaussian function_score to boost the locations that are closer to a given origin.
Finally, these scores are overridable by using a _geo_distance sort where you'd order by the proximity (while of course keeping the match query intact):
...
"query: {...},
"sort": [
{
"_geo_distance": {
"location": {
"lat": 40.7388,
"lon": -73.9982
},
"order": "asc"
}
}
]
}

Bulk geometry ES query

I currently have an ES query to find the nearest location to a lat/long:
GET /geo/_search
{
"sort": [
{
"_geo_distance": {
"geometry": {
"lat": 64,
"lon": 34
},
"order": "asc",
"unit": "mi",
"distance_type": "plane"
}
}
],
"size": 1
}
I want to be able to run this in 1 query for multiple lat/longs, which would return each lat/long related to their nearest location. Is there some way to do this?
GET /geo/_search
{
"sort": [
{
"_geo_distance": {
"geometry": [
{
"lat": 64,
"lon": 34
},
{
"lat": 0,
"lon": 0
}
],
"order": "asc",
"unit": "mi",
"distance_type": "plane"
}
}
],
"size": 1
}
This answer gives more than one point per geo_point. I don't think you can retrieve only the top one per geo_point in one query. You might need to filter the results or to use a loop per each geo_point

Elasticsearch apply condintions in query on basis of results count

Is there any way in Elasticsearch for following type of outcome
"Apply first condition, if no results found then apply next conditions and so on.."
I am aware of basics of ES queries. I know this can be done by querying again and again on results basis but I want to do this in single query for the sake of time and efficiency.
Here is my current query
GET_search{
"query": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"geo_bounding_box": {
"location": {
"top_left": {
"lat": 28.6143519,
"lon": -81.50773
},
"bottom_right": {
"lat": 28.3479859,
"lon": -81.22977
}
}
}
}
]
}
}
}
},
"size": 10,
"from": 0,
"sort": {
"search_score": {
"order": "desc"
}
}
}
Now what I want to do is, if this query hits zero results then this should search for another increased set of lat lon bounds. I can do this by requering elasticsearch but it will be an inefficient way.
I want to know if is this possible in elasticsearch?

Elasticsearch geo_distance filter - distance unit set to km but seems to calculate with meters

I'm using geo_distance filter to get the points that are within a certain distance to a reference point. As a distance_unit I set km, however, when I'm running my code I'm pretty sure that Elastic calculates meters, and not kilometers (so in this particular case below I get the points that are within 20 meters, not 20 kilometers).
Any ideas why is it using meters rather than kilometers?
{
"query": {
"filtered": {
"query": {
"bool": {
"must_not": [
{
"term": {
"_id": {
"value": idnum
}
}
}
]
}
},
"filter": {
"geo_distance": {
"distance": 20,
"distance_unit": "km",
"geopoint": {
"lat": lat,
"lon": lng
}
}
}
}
}
}
use "distance": "20KM", rather than distance_unit....

Resources