How do I go about fetching all documents w/o any objects in a field?
I have the following mapping:
"properties" : {
"name": {
"type": "text"
},
"nestedArray": {
"type": "nested",
"properties": {
"prop1": {
"type": "text"
},
"prop2": {
"type": "text"
}
}
}
}
and I want to get all documents where "nestedArray" is empty or doesn't exist.
I'm using elasticSearch 5.0
I think exists query would solve this problem. Try following query
{
"query": {
"bool": {
"must_not": [
{
"nested": {
"path": "nestedArray",
"query": {
"bool": {
"filter": {
"exists": {
"field": "nestedArray"
}
}
}
}
}
}
]
}
}
}
Try this
{
"size": 5000,
"query": {
"bool": {
"filter": [],
"must_not": [
{
"nested": {
"path": "nestedArray",
"query": {
"exists": {
"field": "nestedArray"
}
}
}
}
]
}
},
"from": 0
}
Related
i have index in es and this is the corresponding mapping :
'''
GET /vid_detect2/_mapping
{
"properties": {
"date":{"type":"date"},
"time":{ "type": "text",
"fielddata": true},
"frame_id": {"type":"integer"},
"camera_id":{"type":"integer"},
"path":{"type":"text"},
"objects" : {"type": "nested",
"properties": {
"class": { "type": "text" ,"fielddata":true },
"confidence": { "type": "float" },
"coordinates":{ "type": "nested" ,
"properties": { "x" :{"type":"float"},
"y" :{"type":"float"},
"w" :{"type":"float"},
"h" :{"type":"float"}
} }
}
}
}
}'''
I want to run following query first :
"query": {
"bool": {
"must": [
{
"nested": {
"path": "objects",
"query": {
"bool": {
"must": [
{ "match": { "objects.class": "person" }}
]
}}}}
]
}}
and then aggregate the returned results with respect to camera_id and further aggregate those aggregated results with date histogram. Please help.
Good start! You can now simply add an aggregation section to achieve what you want:
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "objects",
"query": {
"bool": {
"must": [
{
"match": {
"objects.class": "person"
}
}
]
}
}
}
}
]
}
},
"aggs": {
"camera": {
"terms": {
"field": "camera_id"
},
"aggs": {
"histo": {
"date_histogram": {
"field": "date",
"interval": "day"
}
}
}
}
}
}
I'm try to extract aggregated data, but I'm a little lost when I want to further filter a set of documents. Getting the color seems ok, but when I want to aggregate the categories with some colors filter the query fail. What am I doing wrong on this query?
This is the query I already have:
GET/my_index/_search
{
"_source": false,
"aggs": {
"global": {
"global": {
},
"aggs": {
"all_products": {
"nested": {
"path": "simple"
},
"aggs": {
"filter_top": {
"filter": {
"bool": {
"must": [
{
"match": {
"simple.compound_words": {
"query": "tisch",
"operator": "AND"
}
}
}
]
}
},
"aggs": {
"filter_merged": {
"aggs": {
"filter": {
"bool": {
"must": [
{
"terms": {
"simple.filter_color": [
"green",
"red"
]
}
}
]
}
},
"aggs": {
"filter_category": {
"terms": {
"field": "simple.filter_category"
}
}
}
}
},
"filter_color": {
"terms": {
"field": "simple.filter_color"
}
}
}
}
}
}
}
}
}
}
This is the relevant part of the index mappings.
{
"my_index": {
"mappings": {
"_doc": {
"properties": {
"simple": {
"type": "nested",
"properties": {
"compound_words": {
"type": "text",
"analyzer": "GermanCompoundWordsAnalyzer"
},
"filter_category": {
"type": "keyword"
},
"filter_color": {
"type": "keyword"
}
}
}
}
}
}
}
}
Thanks for your support.
I want to filter the document by nested field value. My document is like this and I want to filter it by Color parameter:
{
"_index": "myindex",
"_type": "product",
"_id": "984984",
"_source": {
"id": "98418",
"name": "Product1",
..
"parameters": {
"Color": [
"Black",
"Gold"
]
}
}
}
My mapping is:
{
"myindex": {
"mappings": {
"product": {
"properties": {
..
"parameters": {
"type": "nested",
"properties": {
"Color": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
..
}
}
}
}
}
}
And my filter query is following:
{
"query": {
"bool": {
"filter": {
"nested": {
"path": "parameters",
"query": {
"term": {
"parameters.Color":"Gold"
}
}
}
}
}
}
}
But unfortunatelly I'm getting zero documents and I do not understand why?
Thank you
EDIT
Event this is working:
{
"query": {
"nested": {
"path": "parameters",
"query": {
"bool": {
"must": [
{ "match": { "parameters.Color": "Gold" }}
]
}
}
}
}
}
..but this is not:
{
"query": {
"nested": {
"path": "parameters",
"query": {
"bool": {
"filter": [
{ "term": { "parameters.Color": "Gold" }}
]
}
}
}
}
}
Why??
Your term query is looking for an exact match and the match query is analyzed before searching. If you are using the standard analyzer it will lowercase your terms when it analyzes them.
You could use the keyword field if you need to do an exact match.
{
"query": {
"bool": {
"filter": {
"nested": {
"path": "parameters",
"query": {
"term": {
"parameters.Color.keyword":"Gold"
}
}
}
}
}
}
}
My mapping is:
"properties": {
"user": {
"type": "nested",
"properties": {
"id": {
"type": "integer"
},
"is_active": {
"type": "boolean",
"null_value": false
},
"username": {
"type": "string"
}
}
},
I want to get all documents that do not have a user field.
I tried:
GET /index/type/_search
{
"query": {
"bool": {
"must_not": [
{
"exists": {
"field": "user"
}
}
]
}
}
}
Which returns all documents.
Based on ElasticSearch 2.x exists filter for nested field doesn't work, I also tried:
GET /index/type/_search
{
"query": {
"nested": {
"path": "user",
"query": {
"bool": {
"must_not": [
{
"exists": {
"field": "user"
}
}
]
}
}
}
}
}
Which returns 0 documents.
What is the correct query to get all documents missing the user field?
I found the correct syntax, it should have been:
GET /index/type/_search
{
"query": {
"bool": {
"must_not": [
{
"nested": {
"path": "user",
"query": {
"exists": {
"field": "user"
}
}
}
}
]
}
}
}
Try using the parent of the user, here obj
GET users/users/_search
{
"query": {
"bool": {
"must_not": [
{
"exists": {
"field": "obj.user"
}
}
]
}
}
}
we have following structure in an index - following is only a partial and doc relevant for this question.
"instance" : {
"id" : 1,
{"instFields": [
{
"sourceFieldId": 2684,
"fieldValue": "false",
"fieldBoolean": false
},
{
"sourceFieldId": 1736,
"fieldValue": "DODGE",
"fieldString": "DODGE"
},
{
"sourceFieldId": 1560,
"fieldValue": "GRAY",
"fieldString": "GRAY"
},
{
"sourceFieldId": 1558,
"fieldValue": "CHALLENGER",
"fieldString": "CHALLENGER"
},
{
"sourceFieldId": 1556,
"fieldValue": "2010",
"fieldDouble": 2010
}
]
}
first user query is give me all instances where sourceFieldId=1736 - this returns all the DODGE instances[] - all this is working fine with an appripriate Elastic Search query. now when user is seeing all DODGE records - user wants to sort by any of those sourceFieldIds for e.g. say user is wanting to sort results by - color - sourceFieldId=1560.
say we have following sort query
{
"query": {
"bool": {
"filter": {
"bool": {
"must": [
{
"nested": {
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"bool": {
"must": [
{
"term": {
"instance.dataSourceId": "196"
}
},
{
"term": {
"instance.dsTypeId": "5"
}
},
{
"nested": {
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"bool": {
"must": [
{
"term": {
"instance.instFields.sourceFieldId": "1558"
}
},
{
"term": {
"instance.instFields.fieldString.raw": "challenger"
}
}
]
}
}
}
},
"path": "instance.instFields"
}
}
]
}
}
}
},
"path": "instance",
"inner_hits": {
"name": "inner_data"
}
}
},
{
"nested": {
"query": {
"bool": {
"should": {
"bool": {
"must": [
{
"match": {
"instance.entitlements.roleId": {
"query": "1",
"type": "boolean"
}
}
},
{
"match": {
"instance.entitlements.read": {
"query": "true",
"type": "boolean"
}
}
}
]
}
}
}
},
"path": "instance.entitlements"
}
}
]
}
}
}
},
"sort": {
"instance.instFields.fieldString.raw": {
"order": "asc",
"nested_path": "instance.instFields",
"nested_filter": {
"bool": {
"filter": {
"bool": {
"must": [
{
"nested": {
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"bool": {
"must": [
{
"term": {
"instance.dataSourceId": "196"
}
},
{
"term": {
"instance.dsTypeId": "5"
}
},
{
"nested": {
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"bool": {
"must": [
{
"term": {
"instance.instFields.sourceFieldId": "1558"
}
},
{
"term": {
"instance.instFields.fieldString.raw": "challenger"
}
}
]
}
}
}
},
"path": "instance.instFields"
}
}
]
}
}
}
},
"path": "instance",
"inner_hits": {
"name": "inner_data1"
}
}
},
{
"nested": {
"query": {
"bool": {
"should": {
"bool": {
"must": [
{
"match": {
"instance.entitlements.roleId": {
"query": "1",
"type": "boolean"
}
}
},
{
"match": {
"instance.entitlements.read": {
"query": "true",
"type": "boolean"
}
}
}
]
}
}
}
},
"path": "instance.entitlements"
}
}
]
}
}
}
}
}
}
}
resulting docs must return entire instance with all the soureceFields - as on a user page it displays other values of DODGE as well.
now issue is- sort query still has to have knowledge to sort where - "sourceFieldId": 1560 (which is a sourceFieldId for color) to sort on color
is there a way to achieve such a sort query in ES without using dynamic scripting/dynamic templating? something like
"sort": {
"instance.instFields.fieldString.raw": (where sourceFieldId=1560?)
Should be able to achieve this using nested_filter option in sort
From the documentation:
nested_filter A filter that the inner objects inside the nested path
should match with in order for its field values to be taken into
account by sorting. Common case is to repeat the query / filter inside
the nested filter or query. By default no nested_filter is active.
For example to sort on color field it would be:
{
"sort": {
"instance.instFields.fieldValue.raws": {
"order": "asc",
"nested_path": "instance.instFields",
"nested_filter": {
"term": {
"instance.instFields.sourceFieldId": "1560"
}
}
}
}
}
Edited
"sort": [{
"instance.instFields.fieldValue": {
"order": "asc",
"nested_path": "instance.instFields",
"nested_filter": {
"term": {
"instance.instFields.sourceFieldId": "1560"
}
}
}
},
{
"instance.instFields.fieldValue": {
"order": "asc",
"nested_path": "instance.instFields",
"nested_filter": {
"term": {
"instance.instFields.sourceFieldId": "1558"
}
}
}
}
]