Get all docs which not contains the key? - elasticsearch

For example,i have 2 type docs,such as
{
"field2":"xx",
"field1","x"
}
{
"field1","x"
}
The one has 2 fields(field1 and field2),another one just has 1 field(field1).
Now,i want to query all docs which do not have field2 field?
EIDT
dsl:
{
"query": {
"bool": {
"filter": [
{
"exists": {
"field": "LableToMember"
}
}
]
}
}
}
doc:
{
"LableToMember": [
{
"xxx": "xxx",
"id": "1"
}
],
"field2":"xxx"
}
LableToMember is a nested field.I find exists api can't be used for nested field?


Note that in ES 5.x the missing query has been removed in favor of the exists one.
So if you want to be forward compatible, you should prefer using this:
POST /_search
{
"query": {
"bool": {
"must_not": {
"exists": {
"field": "field2"
}
}
}
}
}
UPDATE
If you want to retrieve all docs which don't have field2 or have field2 with a given value, you can do it like this:
POST /_search
{
"query": {
"bool": {
"minimum_should_match": 1,
"should": [
{
"bool": {
"must_not": {
"exists": {
"field": "field2"
}
}
}
},
{
"term": {
"field2": "somevalue"
}
}
]
}
}
}


In short you want to query those documents which have field2 missing. You can use Missing Query like:
"filter" : {
"missing" : { "field" : "field2" }
}
Hope it helps

Related

Elastic search combine match and should

I want to create a query where i can use match, and that match should only consider rows where another field value is null or some specific value.
So something like:
where field1 like 'string' and field2 is null or field2 = 123
Is that possible in JS client?

You can use a combination of bool/must/should along with the exists and wildcard query
{
"query": {
"bool": {
"must": [
{
"wildcard": {
"field1": "string*"
}
},
{
"bool": {
"should": [
{
"term": {
"field2": 123
}
},
{
"bool": {
"must_not": {
"exists": {
"field": "field2"
}
}
}
}
]
}
}
]
}
}
}

Elasticsearch: Multiple fields, return true if any exists

ES version: 5.2 alpine
I have a document like this:
{
"field1": null,
"field2": "xyz"
"field3": null
}
I want to return the document if any of the fields above exists/not null.
"filter": {
"bool": {
"must": [
{
"exists": {
"field": ["field1","field2"]
}
}]
}
}
but I get following error.
[exists] unknown token [START_ARRAY] after [field]
Any idea how to do so with this ES version?
Thanks.

You cannot pass multiple fields in exists query. Use exists query for each field and wrap them in should clause as below:
{
"query": {
"bool": {
"should": [
{
"exists": {
"field": "field1"
}
},
{
"exists": {
"field": "field2"
}
},
{
"exists": {
"field": "field3"
}
}
]
}
}
}

I am not sure you want to return the doc if field value exist, or also the value. This is for returning documents which satisfies the condition.
This can help,
You can use should query with minimum should match as 1.
"filter": {
"bool": {
"should": [
{ "exists": { "field": "field1 } },
{ "exists": { "field": "field2 } },
],
"minimum_should_match" : 1,
}
}
Try it, I hope it will work in your version as well.

How to filter a specific value within a dictionary?

Let's say I have this dictionary:
{
"name": "Jorje",
"surname": "Costali",
"extra_information": {
"real_name": "mamino",
"fake_name": "bambino",
"age": "43",
"gang": "gang34"
}
}
How can I query to get all entries that have "extra_information.gang":"gang34" ? I would like to know how to filter after exact term or having a match.
I have tried:
{
"size": 20,
"query": {
"bool": {
"filter": [
{
"terms": {
"extra_information.gang": [
"gang34"
]
}
}
]
}
}
}
but it does not return any entries.
I have tried:
GET _search
{
"query": {
"bool": {
"must": [
{
"match": {
"extra_information.gang" : "gang34"
}
}
]
}
}
}
and works, but I want to make it into a filter, not a simple match query.

Did you try to use .keyword? like:
"terms": {
"extra_information.gang.keyword": [
"gang34"
]
}
I tried what you wrote on my nested dictionary document, it works like this to me.

How to add multiple exists fields with OR operation in elasticsearch

I have the following query:
{
"query": {
"exists": {
"field": "field1"
}
}
}
I would like my query to be modified in a way that I could add another filter with existing field having 'OR' operation between them.
Any idea how to do so?
Thanks.

You can use Bool Query with should.
{
"query": {
"bool": {
"should": [{
"exists": {
"field": "field1"
}
},
{
"exists": {
"field": "field2"
}
}
]
}
}
}

Use should Bool Query if either field1 OR field2 should exist from the resulting documents.
Example:
POST your_index/_doc/_search
{
"query": {
"bool": {
"should": [{
"exists": {
"field": "field1"
}
},
{
"exists": {
"field": "field2"
}
}
]
}
}
}
Use must Bool Query if either field1 AND field2 must exist in the output documents.
Example:
POST your_index/_doc/_search
{
"query": {
"bool": {
"must": [{
"exists": {
"field": "field1"
}
},
{
"exists": {
"field": "field2"
}
}
]
}
}
}

Create Elasticsearch curl query for not null and not empty("")

How can i create Elasticsearch curl query to get the field value which are not null and not empty(""),
Here is the mysql query:
select field1 from mytable where field1!=null and field1!="";

A null value and an empty string both result in no value being indexed, in which case you can use the exists filter
curl -XGET 'http://127.0.0.1:9200/test/test/_search?pretty=1' -d '
{
"query" : {
"constant_score" : {
"filter" : {
"exists" : {
"field" : "myfield"
}
}
}
}
}
'
Or in combination with (eg) a full text search on the title field:
curl -XGET 'http://127.0.0.1:9200/test/test/_search?pretty=1' -d '
{
"query" : {
"filtered" : {
"filter" : {
"exists" : {
"field" : "myfield"
}
},
"query" : {
"match" : {
"title" : "search keywords"
}
}
}
}
}
'

As #luqmaan pointed out in the comments, the documentation says that the filter exists doesn't filter out empty strings as they are considered non-null values.
So adding to #DrTech's answer, to effectively filter null and empty string values out, you should use something like this:
{
"query" : {
"constant_score" : {
"filter" : {
"bool": {
"must": {"exists": {"field": "<your_field_name_here>"}},
"must_not": {"term": {"<your_field_name_here>": ""}}
}
}
}
}
}

On elasticsearch 5.6, I have to use command below to filter out empty string:
GET /_search
{
"query" : {
"regexp":{
"<your_field_name_here>": ".+"
}
}
}

Wrap a Missing Filter in the Must-Not section of a Bool Filter. It will only return documents where the field exists, and if you set the "null_value" property to true, values that are explicitly not null.
{
"query":{
"filtered":{
"query":{
"match_all":{}
},
"filter":{
"bool":{
"must":{},
"should":{},
"must_not":{
"missing":{
"field":"field1",
"existence":true,
"null_value":true
}
}
}
}
}
}
}

You can do that with bool query and combination of must and must_not like this:
GET index/_search
{
"query": {
"bool": {
"must": [
{"exists": {"field": "field1"}}
],
"must_not": [
{"term": {"field1": ""}}
]
}
}
}
I tested this with Elasticsearch 5.6.5 in Kibana.

The only solution here that worked for me in 5.6.5 was bigstone1998's regex answer. I'd prefer not to use a regex search though for performance reasons. I believe the reason the other solutions don't work is because a standard field will be analyzed and as a result have no empty string token to negate against. The exists query won't help on it's own either since an empty string is considered non-null.
If you can't change the index the regex approach may be your only option, but if you can change the index then adding a keyword subfield will solve the problem.
In the mappings for the index:
"myfield": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
}
Then you can simply use the query:
{
"query": {
"bool": {
"must": {
"exists": {
"field": "myfield"
}
},
"must_not": {
"term": {
"myfield.keyword": ""
}
}
}
}
}
Note the .keyword in the must_not component.

You can use not filter on top of missing.
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"not": {
"filter": {
"missing": {
"field": "searchField"
}
}
}
}
}
}

Here's the query example to check the existence of multiple fields:
{
"query": {
"bool": {
"filter": [
{
"exists": {
"field": "field_1"
}
},
{
"exists": {
"field": "field_2"
}
},
{
"exists": {
"field": "field_n"
}
}
]
}
}
}

You can use a bool combination query with must/must_not which gives great performance and returns all records where the field is not null and not empty.
bool must_not is like "NOT AND" which means field!="", bool must exist means its !=null.
so effectively enabling: where field1!=null and field1!=""
GET IndexName/IndexType/_search
{
"query": {
"bool": {
"must": [{
"bool": {
"must_not": [{
"term": { "YourFieldName": ""}
}]
}
}, {
"bool": {
"must": [{
"exists" : { "field" : "YourFieldName" }
}]
}
}]
}
}
}
ElasticSearch Version:
"version": {
"number": "5.6.10",
"lucene_version": "6.6.1"
}

ES 7.x
{
"_source": "field",
"query": {
"bool": {
"must": [
{
"exists": {
"field":"field"
}
}
],
"must_not": [
{
"term": {
"field.keyword": {
"value": ""
}
}
}
]
}
}
}

We are using Elasticsearch version 1.6 and I used this query from a co-worker to cover not null and not empty for a field:
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"bool": {
"must": [
{
"exists": {
"field": "myfieldName"
}
},
{
"not": {
"filter": {
"term": {
"myfieldName": ""
}
}
}
}
]
}
}
}
}
}

You need to use bool query with must/must_not and exists
To get where place is null
{
"query": {
"bool": {
"must_not": {
"exists": {
"field": "place"
}
}
}
}
}
To get where place is not null
{
"query": {
"bool": {
"must": {
"exists": {
"field": "place"
}
}
}
}
}

Elastic search Get all record where condition not empty.
const searchQuery = {
body: {
query: {
query_string: {
default_field: '*.*',
query: 'feildName: ?*',
},
},
},
index: 'IndexName'
};

Resources