elastic exists query for nested documents - elasticsearch

I have a nested documents as:
"someField": "hello",
"users": [
{
"name": "John",
"surname": "Doe",
"age": 2
}
]
according to this https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-exists-query.html, the above should match:
GET /_search
{
"query": {
"exists" : { "field" : "users" }
}
}
whereas the following should not,
"someField": "hello",
"users": []
but unfortunately both do not match. any ideas?


The example mentioned on the Elasticsearch blog refers to string and array of string types, not for nested types.
The following query should work for you:
{
"query": {
"nested": {
"path": "users",
"query": {
"bool": {
"must": [
{
"exists": {
"field": "users"
}
}
]
}
}
}
}
}
Also, you can refer to this issue for more info, which discusses this usage pattern.


This works for me
GET /type/_search?pretty=true
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "outcome",
"query": {
"exists": {
"field": "outcome.outcomeName"
}
}
}
}
]
}
}
}


With the following index mapping:
{
"index_name": {
"mappings": {
"object_name": {
"dynamic": "strict",
"properties": {
"nested_field_name": {
"type": "nested",
"properties": {
"some_property": {
"type": "keyword"
}
}
}
}
}
}
}
}
I needed to use this query:
GET /index_name/_search
{
"query": {
"nested": {
"path": "nested_field_name",
"query": {
"bool": {
"must": [
{
"exists": {
"field": "nested_field_name.some_property"
}
}
]
}
}
}
}
}
Elasticsearch version 5.4.3


The answer from user3775217 has worked for me but I needed to tweak it to work as expected for must_not. Essentially the bool/must needed to be wrapped around the nested portion of the query:
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "users",
"query": {
"exists": {
"field": "users"
}
}
}
}
}
]
}
}

Related

How can I put `must_not` under `filter` in Elasticsearch?

I'd like to use not equal in my filter but it doesn't work [13:24] [bool] failed to parse field [filter]:
"query": {
"bool": {
"filter": [
{
"must_not" : {
"term" : {
"status" : "DECLINED"
}
}
},
{
"term": { "type": "ORDER"}
}
]
}
}
it works if I put the must_not under query like below. How can I put not equal in filter?
"query": {
"bool": {
"must_not": {
"term": {
"status": "DECLINED"
}
},
"filter": ...

May be one more bool needed inside the filter ?
/_search
{
"query": {
"bool": {
"filter": {
"bool": {
"must": [
{
"term": { "type": "ORDER"}
}
],
"must_not": [
{
"term": {
"status": "DECLINED"
}
}
]
}
}
}
}
}

ElasticSearch should with nested and bool must_not exists

With the following mapping:
"categories": {
"type": "nested",
"properties": {
"category": {
"type": "integer"
},
"score": {
"type": "float"
}
}
},
I want to use the categories field to return documents that either:
have a score above a threshold in a given category, or
do not have the categories field
This is my query:
{
"query": {
"bool": {
"should": [
{
"nested": {
"path": "categories",
"query": {
"bool": {
"must": [
{
"terms": {
"categories.category": [
<id>
]
}
},
{
"range": {
"categories.score": {
"gte": 0.5
}
}
}
]
}
}
}
},
{
"bool": {
"must_not": [
{
"exists": {
"field": "categories"
}
}
]
}
}
],
"minimum_should_match": 1
}
}
}
It correctly returns documents both with and without the categories field, and orders the results so the ones I want are first, but it doesn't filter the results having score below the 0.5 threshold.

Great question.
That is because categories is not exactly a field from the elasticsearch point of view[a field on which inverted index is created and used for querying/searching] but categories.category and categories.score is.
As a result categories being not found in any document, which is actually true for all the documents, you observe the result what you see.
Modify the query to the below and you'd see your use-case working correctly.
POST <your_index_name>/_search
{
"query": {
"bool": {
"should": [
{
"nested": {
"path": "categories",
"query": {
"bool": {
"must": [
{
"terms": {
"categories.category": [
"100"
]
}
},
{
"range": {
"categories.score": {
"gte": 0.5
}
}
}
]
}
}
}
},
{
"bool": {
"must_not": [ <----- Note this
{
"nested": {
"path": "categories",
"query": {
"bool": {
"must": [
{
"exists": {
"field": "categories.category"
}
},
{
"exists": {
"field": "categories.score"
}
}
]
}
}
}
}
]
}
}
],
"minimum_should_match": 1
}
}
}

Score keyword terms query on nested fields in elastichsearch 6.3

I have a set of keywords (skills in my example) and I would like to retrieve documents which match most of them. The documents should be sorted by how many of the keywords they match. The field i am searching into (skills) is of nested type. The index has the following mapping:
{
"mappings": {
"profiles": {
"properties": {
"id": {
"type": "keyword"
},
"skills": {
"type": "nested",
"properties": {
"level": {
"type": "float"
},
"name": {
"type": "keyword"
}
}
}
}
}
}
}
I tried both a terms query on the keyword field like:
{
"query": {
"nested": {
"path": "skills",
"query": {
"terms": {
"skills.name": [
"python",
"java"
]
}
}
}
}
}
And a boolean query
{
"query": {
"nested": {
"path": "skills",
"query": {
"bool": {
"should": [
{
"terms": {
"skills.name": [
"java"
]
}
},
{
"terms": {
"skills.name": [
"r"
]
}
}
]
}
}
}
}
}
For both queries the maximum score of the returned documents is 1. Thus both return documents that have ANY of the skills, but do not sort them such those with both skills are on top. The issues seems to be that skills is a nested field.

The second query works if each element of should is a nested query.
{
"query": {
"bool": {
"should": [
{
"nested": {
"path": "skills",
"query": {
"terms": {
"skills.name": [
"java"
]
}
}
}
},
{
"nested": {
"path": "skills",
"query": {
"terms": {
"skills.name": [
"r"
]
}
}
}
}
]
}
}
}

elasticsearch query nested array of objects

Hi I am trying to get a query to filter based on values in an array of objects, the structure is like this
{
"_index": "test",
"_type": "home",
"_id": "1247816",
"_score": 1,
"_source": {
"TranCust": {
"CustId": 1247816,
"sourceNodeName": "SRC"
},
"TranList": [
{
"TranId": 2431015,
"batchNr": "211"
},
{
"TranId": 2431016,
"batchNr": "213"
}
]
}
}
as an example, i would like to find all documents with a TranId of 2431015, my query looks like this
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "TranList",
"query": {
"bool": {
"must": [
{
"match": {
"TranId": "2431015"
}
}
]
}
}
}
}
]
}
}
}
it seems to return no results, is there a better way to try and write this query ?
EDIT,
here are the mappings put in
{
"mappings": {
"home": {
"properties": {
"TranCust": {
"type": "object"
}
},
"TranList": {
"type": "nested"
}
}
}
}
}

ok, so after lots of attempts this is how i got it to work
{
"query": {
"bool": {
"must": [{
"nested": {
"path": "TranList",
"query": {
"bool": {
"must": [{
"match": {
"TranList.TranId": "2431015"
}
}]
}
}
}
}]
}
}
}

Not sure what was your ES version, but the following should ideally work for ES 6.x+ versions. You don't actually need to wrap your nested query with bool:must
{
"query": {
"nested" : {
"path" : "TranList",
"query" : {
"bool" : {
"must" : [
{ "match" : {"TranList.TranId" : "2431015"} }
]
}
}
}
}
}

{
"query": {
"query_string": {
"default_field": "TranList.TranId",
"query": "2431015"
}
}
}

Create Elasticsearch curl query for not null and not empty("")

How can i create Elasticsearch curl query to get the field value which are not null and not empty(""),
Here is the mysql query:
select field1 from mytable where field1!=null and field1!="";

A null value and an empty string both result in no value being indexed, in which case you can use the exists filter
curl -XGET 'http://127.0.0.1:9200/test/test/_search?pretty=1' -d '
{
"query" : {
"constant_score" : {
"filter" : {
"exists" : {
"field" : "myfield"
}
}
}
}
}
'
Or in combination with (eg) a full text search on the title field:
curl -XGET 'http://127.0.0.1:9200/test/test/_search?pretty=1' -d '
{
"query" : {
"filtered" : {
"filter" : {
"exists" : {
"field" : "myfield"
}
},
"query" : {
"match" : {
"title" : "search keywords"
}
}
}
}
}
'

As #luqmaan pointed out in the comments, the documentation says that the filter exists doesn't filter out empty strings as they are considered non-null values.
So adding to #DrTech's answer, to effectively filter null and empty string values out, you should use something like this:
{
"query" : {
"constant_score" : {
"filter" : {
"bool": {
"must": {"exists": {"field": "<your_field_name_here>"}},
"must_not": {"term": {"<your_field_name_here>": ""}}
}
}
}
}
}

On elasticsearch 5.6, I have to use command below to filter out empty string:
GET /_search
{
"query" : {
"regexp":{
"<your_field_name_here>": ".+"
}
}
}

Wrap a Missing Filter in the Must-Not section of a Bool Filter. It will only return documents where the field exists, and if you set the "null_value" property to true, values that are explicitly not null.
{
"query":{
"filtered":{
"query":{
"match_all":{}
},
"filter":{
"bool":{
"must":{},
"should":{},
"must_not":{
"missing":{
"field":"field1",
"existence":true,
"null_value":true
}
}
}
}
}
}
}

You can do that with bool query and combination of must and must_not like this:
GET index/_search
{
"query": {
"bool": {
"must": [
{"exists": {"field": "field1"}}
],
"must_not": [
{"term": {"field1": ""}}
]
}
}
}
I tested this with Elasticsearch 5.6.5 in Kibana.

The only solution here that worked for me in 5.6.5 was bigstone1998's regex answer. I'd prefer not to use a regex search though for performance reasons. I believe the reason the other solutions don't work is because a standard field will be analyzed and as a result have no empty string token to negate against. The exists query won't help on it's own either since an empty string is considered non-null.
If you can't change the index the regex approach may be your only option, but if you can change the index then adding a keyword subfield will solve the problem.
In the mappings for the index:
"myfield": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
}
Then you can simply use the query:
{
"query": {
"bool": {
"must": {
"exists": {
"field": "myfield"
}
},
"must_not": {
"term": {
"myfield.keyword": ""
}
}
}
}
}
Note the .keyword in the must_not component.

You can use not filter on top of missing.
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"not": {
"filter": {
"missing": {
"field": "searchField"
}
}
}
}
}
}

Here's the query example to check the existence of multiple fields:
{
"query": {
"bool": {
"filter": [
{
"exists": {
"field": "field_1"
}
},
{
"exists": {
"field": "field_2"
}
},
{
"exists": {
"field": "field_n"
}
}
]
}
}
}

You can use a bool combination query with must/must_not which gives great performance and returns all records where the field is not null and not empty.
bool must_not is like "NOT AND" which means field!="", bool must exist means its !=null.
so effectively enabling: where field1!=null and field1!=""
GET IndexName/IndexType/_search
{
"query": {
"bool": {
"must": [{
"bool": {
"must_not": [{
"term": { "YourFieldName": ""}
}]
}
}, {
"bool": {
"must": [{
"exists" : { "field" : "YourFieldName" }
}]
}
}]
}
}
}
ElasticSearch Version:
"version": {
"number": "5.6.10",
"lucene_version": "6.6.1"
}

ES 7.x
{
"_source": "field",
"query": {
"bool": {
"must": [
{
"exists": {
"field":"field"
}
}
],
"must_not": [
{
"term": {
"field.keyword": {
"value": ""
}
}
}
]
}
}
}

We are using Elasticsearch version 1.6 and I used this query from a co-worker to cover not null and not empty for a field:
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"bool": {
"must": [
{
"exists": {
"field": "myfieldName"
}
},
{
"not": {
"filter": {
"term": {
"myfieldName": ""
}
}
}
}
]
}
}
}
}
}

You need to use bool query with must/must_not and exists
To get where place is null
{
"query": {
"bool": {
"must_not": {
"exists": {
"field": "place"
}
}
}
}
}
To get where place is not null
{
"query": {
"bool": {
"must": {
"exists": {
"field": "place"
}
}
}
}
}

Elastic search Get all record where condition not empty.
const searchQuery = {
body: {
query: {
query_string: {
default_field: '*.*',
query: 'feildName: ?*',
},
},
},
index: 'IndexName'
};

Resources