Apply geo distance filter on nested field - elasticsearch

My mapping contains a nested field like this:
"Locations": {
"type": "nested",
"properties": {
"Name": {
"type": "string"
},
"GeoPoint": {
"type": "geo_point"
}
}
}
So basically what I'm trying to do is store some additional attributes with every location of the document.
Unfortunately, it looks like this won't work with a geo distance filter:
GET /myIndex/myType/_search
{
"filter": {
"geo_distance": {
"distance": "100 mi",
"Locations.GeoPoint": {
lat: 40.70,
lon: -74.00
}
}
}
}
won't return the any results, whereas the filter works flawlessly if the GeoPoint is directly on the document itself instead of on the nested field:
GET /myIndex/myType/_search
{
"filter": {
"geo_distance": {
"distance": "100 mi",
"GeoPoint": {
lat: 40.70,
lon: -74.00
}
}
}
}
Is there any way to make geo distance filter work with geo_point on the nested field?

A nested filter does the job:
GET /myIndex/myType/_search
{
"filter" : {
"nested" : {
"path" : "Locations",
"filter" : {
"geo_distance": {
"distance": "100 mi",
"GeoPoint": {
"lat": 40.70,
"lon": -74.00
}
}
}
}
}
}

An update for ElasticSearch 5.5 users, #Max's answer has been deprecated for the Nested Filter and Nested Query feature using the dot notation instead of explicitly wrapping a "nested" field around your query.
For example:
{
"query": {
"nested" : {
"path" : "obj1",
"score_mode" : "avg",
"query" : {
"bool" : {
"must" : [
{ "match" : {"obj1.name" : "blue"} },
{ "range" : {"obj1.count" : {"gt" : 5}} }
]
}
}
}
}
}
More information: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-nested-filter.html

Related

Place an Analyzer on a a specific array item in a nested object

I have the following mapping
"mappings":{
"properties":{
"name": {
"type": "text"
},
"age": {
"type": "integer"
},
"customProps":{
"type" : "nested",
"properties": {
"key":{
"type": "keyword"
},
"value": {
"type" : "keyword"
}
}
}
}
}
example data
{
"name" : "person1",
"age" : 10,
"customProps":[
{"hairColor":"blue"},
{"height":"120"}
]
},
{
"name" : "person2",
"age" : 30,
"customProps":[
{"jobTitle" : "software engineer"},
{"salaryAccount" : "AvGhj90AAb"}
]
}
so i want to be able to search for document by salary account case insensitive, i am also searching using wild card
example query is
{
"query": {
"bool": {
"should": [
{
"nested": {
"path": "customProps",
"query": {
"bool": {
"must": [
{ "match": { "customProps.key": "salaryAccount" } },
{ "wildcard": { "customProps.value": "*AvG*"
}
}
]}}}}]}}}
i tried adding analyzer with PUT using the following syntax
{
"settings":{
"index":{
"analysis":{
"analyzer":{
"analyzer_case_insensitive" : {
"tokenizer":"keyword",
"filter":"lowercase"
}
}
}
}
},
"mappings":{
"people":{
"properties":{
"customProps":{
"properties":{
"value":{
"type": "keyword",
"analyzer": "analyzer_case_insensitive"
}
}
}
}
}
}
}
im getting the following error
"type" : "mapper_parsing_exception",
"reason" : "Root mapping definition has unsupported parameters: [people: {properties={customProps={properties={value={analyzer=analyzer_case_insensitive, type=keyword}}}}}]"
any idea how to do the analyzer for the salary account object in the array when it exists?
Your use case is quite clear, that you want to search on the value of salaryAccount only when this key exists in customProps array.
There are some issues with your mapping definition :
You cannot define a custom analyzer for keyword type field, instead you can use a normalizer
Based on the mapping definition you added at the beginning of the question, it seems that you are using elasticsearch version 7.x. But the second mapping definition that you provided, in that you have added mapping type also (i.e people), which is deprecated in 7.x
There is no need to add the key and value fields in the index mapping.
Adding a working example with index mapping, search query, and search result
Index Mapping:
PUT myidx
{
"mappings": {
"properties": {
"customProps": {
"type": "nested"
}
}
}
}
Search Query:
You need to use exists query, to check whether a field exists or not. And case_insensitive param in Wildcard query is available since elasticsearch version 7.10. If you are using a version below this, then you need to use a normalizer, to achieve case insensitive scenarios.
POST myidx/_search
{
"query": {
"bool": {
"should": [
{
"nested": {
"path": "customProps",
"query": {
"bool": {
"must": [
{
"exists": {
"field": "customProps.salaryAccount"
}
},
{
"wildcard": {
"customProps.salaryAccount.keyword": {
"value": "*aVg*",
"case_insensitive": true
}
}
}
]
}
}
}
}
]
}
}
}
Search Result:
"hits" : [
{
"_index" : "myidx",
"_type" : "_doc",
"_id" : "2",
"_score" : 2.0,
"_source" : {
"name" : "person2",
"age" : 30,
"customProps" : [
{
"jobTitle" : "software engineer"
},
{
"salaryAccount" : "AvGhj90AAb"
}
]
}
}
]

Elasticsearch preform "OR" search on groups of filtering criteria

Overview: I have a situation where within a single index I want to preform a search to return results based on 2 different sets of criteria. Imagine a scenario where where I have an index with a data structure like what is outlined below
I want to preform some sort of query that looks at different "blocks" of criteria. Meaning I want to search by both of the following categories in a single query (if possible):
Category One:
Distance / location
Public = true
--OR--
Category Two:
Distance / location
Public = false
category = "specific category"
(although this is not my exact scenario it is an illustration of what I am facing):
{
"mappings" : {
"properties" : {
"category" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"completed" : {
"type" : "boolean"
},
"deleted" : {
"type" : "boolean"
},
"location" : {
"type" : "geo_point"
},
"public" : {
"type" : "boolean"
},
"uuid" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
Please note: I am rather new to Elastic and would appreciate any help with this. I have attempted to search for this question but was not able to find what I was looking for. Please let me know if there is any missing information here I should include.
Sure, you can combine bool queries
POST test_user2737876/_search
{
"query": {
"bool": {
"should": [
{
"bool": {
"filter": [
{
"geo_distance": {
"distance": "200km",
"location": {
"lat": 41.12,
"lon": -71.34
}
}
},
{
"term": {
"public": false
}
}
]
}
},
{
"bool": {
"filter": [
{
"geo_distance": {
"distance": "200km",
"location": {
"lat": 41.12,
"lon": -71.34
}
}
},
{
"term": {
"category.keyword": "specific category"
}
},
{
"term": {
"public": false
}
}
]
}
}
],
"minimum_should_match": 1
}
}
}

Elasticsearch nested geo-shape query

Suppose I have the following mapping:
"mappings": {
"doc": {
"properties": {
"name": {
"type": "text"
},
"location": {
"type": "nested",
"properties": {
"point": {
"type": "geo_shape"
}
}
}
}
}
}
}
There is one document in the index:
POST /example/doc?refresh
{
"name": "Wind & Wetter, Berlin, Germany",
"location": {
"type": "point",
"coordinates": [13.400544, 52.530286]
}
}
How can I make a nested geo-shape query?
Example of usual geo-shape query from the documentation (the "bool" block can be skipped):
{
"query":{
"bool": {
"must": {
"match_all": {}
},
"filter": {
"geo_shape": {
"location": {
"shape": {
"type": "envelope",
"coordinates" : [[13.0, 53.0], [14.0, 52.0]]
},
"relation": "within"
}
}
}
}
}
}
Example of a nested query is:
{
"query": {
"nested" : {
"path" : "obj1",
"score_mode" : "avg",
"query" : {
"bool" : {
"must" : [
{ "match" : {"obj1.name" : "blue"} },
{ "range" : {"obj1.count" : {"gt" : 5}} }
]
}
}
}
}
}
Now how to combine them? In the documentation it is mentioned that nested filter has been replaced by nested query. And that it behaves as a query in “query context” and as a filter in “filter context”.
If I try query for intersect with the point:
{
"query": {
"nested": {
"path": "location",
"query": {
"geo_shape": {
"location.point": {
"shape": {
"type": "point",
"coordinates": [
13.400544,
52.530286
]
},
"relation": "disjoint"
}
}
}
}
}
}
I still get back the document even if relation is "disjoint", so it's not correct. I tried different combinations, with "bool" and "filter", etc. but query is ignored, returning the whole index. Maybe it's impossible with this type of mapping?
Clearly I am missing something here. Can somebody help me out with that, please? Any help is greatly appreciated.

Filtering on Elasticsearch Optional Fields

I'm using Elasticsearch to query a document type, that has an optional location field. When searching, if that field does not exist, those results should be returned, as well as filtering on the results that do.
It seems like the OR filter in Elasticsearch does not short circuit, as this:
"query": {
"filtered": {
"query": {
"match_phrase_prefix": {
"display_name": "SearchQuery"
}
},
"filter": {
"or": [
{
"missing": {
"field": "location"
}
},
{
"geo_distance" : {
"distance" : "20mi",
"location" : {
"lat" : 33.47,
"lon" : -112.07
}
}
}
]
}
Fails with "failed to find geo_point field [location]".
Is there any way to perform this (or something along the same vein) in ES?
I don't know why yours isn't working but I've used the bool filter with great success in the past. The should option is essentially an or and makes sure at least one is true. Give it a try and comment on my answer if it still doesn't work. Also double check I copied your query terms properly :)
{
"filtered" : {
"query" : {
"match_phrase_prefix": {
"display_name": "SearchQuery"
}
},
"filter" : {
"bool" : {
"should" : [
{
"missing": { "field": "location" }
},
{
"geo_distance" : {
"distance" : "20mi",
"location" : {
"lat" : 33.47,
"lon" : -112.07
}
}
}
]
}
}
}
}
For anyone with the same issue, I kind of just hacked around it. For any documents that were missing a "location", I added one with a lat/lon of 0/0. Then I altered my query to be:
"filter": {
"or": [
{
"geo_distance": {
"distance": "0.1mi",
"location": {
"lat": 0,
"lon": 0
}
}
},
{
"geo_distance": {
"distance": "30mi",
"location": {
"lat": [lat variable],
"lon": [lon variable]
}
}
}
]
}

ElasticSearch how to setup geo_point

I'm trying to setup a geo_point object on ES 1.0.0 and run a simple proof of concept query against it but the query is failing to return any hits. Here are my setup steps:
1) Create the mapping:
PUT jay/geotest/_mapping
{
"geotest" : {
"properties" : {
"name" : {
"type" : "string"
},
"pin" : {
"type": "geo_point"
}
}
}
}
2) verify the mapping:
GET jay/geotest/_mapping
3) Add a piece of data
put jay/geotest/1
{
"name": "test1",
"pin": {
"lat": 0,
"lon": 0
}
}
4) query for that data:
GET jay/geotest/_search?search_type=count
{
"filtered" : {
"filter" : {
"geo_distance" : {
"distance" : "100km",
"pin" : {
"lat" : 0,
"lon" : 0
}
}
}
}
}
My expected result is that I will get one hit returned but instead nothing is returned by the query.
Thanks in advance!
I think you're missing the "query" part of the request.
POST jay/geotest/_search
{
"query": {
"filtered": {
"filter": {
"geo_distance": {
"distance": "100km",
"pin": {
"lat": 0,
"lon": 0
}
}
}
}
}
}
I've just tested your steps, and making that change returns the document.

Resources