How to combine term filters with a missing filter in Elasticsearch? - elasticsearch

We are using Elasticsearch 1.6 and I have a working three term query that I need to modify with a stand alone working missing filter. Here is the current code:
The original term query with three entries
GET ...
{
"query": {
"nested": {
"path": "MAIN_FIELD",
"query": {
"bool": {
"must": [
{
"term": {
"MAIN_FIELD.ID": 1234
}
},
{
"term": {
"MAIN_FIELD.OTHER_IND": "false"
}
},
{
"term": {
"MAIN_FIELD.INDICATOR": "Y"
}
}
]
}
}
}
}
}
The stand alone missing query:
GET ...
{
"query" : {
"filtered" : {
"filter" : {
"missing" : { "field" : "MAIN_FIELD.OTHER_IND" }
}
}
}
}
How do I change the term query from the first query:
"term": {
"MAIN_FIELD.OTHER_IND": "false"
}
to use a missing filter?


I think what you want is below:
{
"query": {
"nested": {
"path": "MAIN_FIELD",
"query": {
"bool": {
"must": [
{
"term": {
"MAIN_FIELD.ID": 1234
}
},
{
"filtered": {
"filter": {
"missing": {
"field": "MAIN_FIELD.OTHER_IND"
}
}
}
},
{
"term": {
"MAIN_FIELD.INDICATOR": "Y"
}
}
]
}
}
}
}
}

Related

How to do AND / OR in Filter Aggregation

In https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-filter-aggregation.html we learn:
{
"aggs" : {
"t_shirts" : {
"filter" : { "term": { "type": "t-shirt" } }
}
}
}
The above code show how we can aggs for t-shirt, but how do we do:
{
"aggs" : {
"t_shirts_or_shorts" : {
"filter" : { "term": { "type": "t-shirt" } OR "term": { "type": "shorts" } }
}
}
}
and
{
"aggs" : {
"black_t_shirts" : {
"filter" : { "term": { "type": "t-shirt" } AND "term": { "color": "black" } }
}
}
}

You can use "must" and should clause in filter aggregation same as in query.
Query1:
{
"size": 0,
"aggs": {
"t_shirts": {
"filter": {
"bool": {
"should": [
{
"term": {
"type.keyword": "t-shirt"
}
},
{
"term": {
"type.keyword": "shorts"
}
}
]
}
}
}
}
}
Query2:
{
"size": 0,
"aggs": {
"t_shirts": {
"filter": {
"bool": {
"must": [
{
"term": {
"type.keyword": "t-shirt"
}
},
{
"term": {
"color.keyword": "black"
}
}
]
}
}
}
}
}

You can combine filter criteria using the bool tag, much like you can query clauses.
You can also check this: Multiple filters and an aggregate in elasticsearch

How to join two queries in one using elasticsearch?

Hi I want to join two queries in one in elasticsearch, but I don't know how to do it: I think I should do an aggregation but I don't know very clear how to do it. Could you help me? My ES version is 5.1.2.
First filter by status and name:
POST test_lite/_search
{
"aggs": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"match": {
"STATUS": "Now"
}
},
{
"match": {
"NAME": "PRUDENTL"
}
}
]
}
}
}
}
}
Look for in the filtered records for the word filtered in description:
POST /test_lite/_search
{
"query": {
"wildcard" : { "DESCRIPTION" : "*english*" }
}
}

The only query needed is:
POST test_lite/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"STATUS": "Now"
}
},
{
"match": {
"NAME": "PRUDENTL"
}
},
{"wildcard" : { "DESCRIPTION" : "*english*" }}
]
}
}
}

How to search both in range and match query in one merged request using Elasticsearch?

I have two assembled queries that work as expected.
First one uses constant score, while matching range between two values:
GET /_search
{
"query" : {
"constant_score" : {
"filter" : {
"range" : {
"locationId" : {
"gte" : 100012138,
"lt" : 101000349
}
}
}
}
}
}
The second one searches for bool.
GET /_search
{
"query": {
"filtered": {
"query": {
"bool": {
"must": [{
"match": {
"name": "Barcelona"
}
}]
}
}
}
}
}
Now I need to merge them and I am struggling how, because tried many combinations of putting in different scopes, but not successful.
So this query returns an error.
GET /_search
{
"query": {
"filtered": {
"query": {
"bool": {
"must": [{
"match": {
"name": "sídlisko"
}
}]
}
}
},
"constant_score" : {
"filter" : {
"range" : {
"locationId" : {
"gte" : 100012138,
"lt" : 1000010349
}
}
}
}
}
}
Error:
... failed to parse search source. expected field name but got
[START_OBJECT]

You could just put constant score query inside bool must clause
{
"query": {
"filtered": {
"query": {
"bool": {
"must": [
{
"match": {
"name": "sidlisko"
}
},
{
"constant_score": {
"filter": {
"range": {
"locationId": {
"gte": 100012138,
"lt": 1000010349
}
}
}
}
}
]
}
}
}
}
}

I've managed to establish this query and it appears to work.
This looks as the most optimised one.
GET /_search
{
"query": {
"filtered": {
"query": {
"bool": {
"must": [{
"match": {
"fullAddress": "sidlisko"
}
}]
}
},
"filter" : {
"range" : {
"locationId" : {
"gte": 100012138,
"lt": 1000010349
}
}
}
}
}
}

ElasticSearch query - exists but doesn't contain

I'm trying to build a query that will return results only if they contain a certain field BUT only if that fields doesn't equal a specific value.
I can't manage the proper syntax:
POST webdata/interaction/_search
{
"query": {
"filtered": {
"filter": {
"exists": {
"field": "mediaType"
},
"and": {
"not" {
"term" : { "mediaType" : "none" }
}
}
}
}
}
}

Use the Bool Filter with must and must_not clauses.
{
"query": {
"filtered": {
"filter": {
"bool": {
"must": {
"exists": {
"field": "mediaType"
}
},
"must_not": {
"term": {
"mediaType": "none"
}
}
}
}
}
}
}

Create Elasticsearch curl query for not null and not empty("")

How can i create Elasticsearch curl query to get the field value which are not null and not empty(""),
Here is the mysql query:
select field1 from mytable where field1!=null and field1!="";

A null value and an empty string both result in no value being indexed, in which case you can use the exists filter
curl -XGET 'http://127.0.0.1:9200/test/test/_search?pretty=1' -d '
{
"query" : {
"constant_score" : {
"filter" : {
"exists" : {
"field" : "myfield"
}
}
}
}
}
'
Or in combination with (eg) a full text search on the title field:
curl -XGET 'http://127.0.0.1:9200/test/test/_search?pretty=1' -d '
{
"query" : {
"filtered" : {
"filter" : {
"exists" : {
"field" : "myfield"
}
},
"query" : {
"match" : {
"title" : "search keywords"
}
}
}
}
}
'

As #luqmaan pointed out in the comments, the documentation says that the filter exists doesn't filter out empty strings as they are considered non-null values.
So adding to #DrTech's answer, to effectively filter null and empty string values out, you should use something like this:
{
"query" : {
"constant_score" : {
"filter" : {
"bool": {
"must": {"exists": {"field": "<your_field_name_here>"}},
"must_not": {"term": {"<your_field_name_here>": ""}}
}
}
}
}
}

On elasticsearch 5.6, I have to use command below to filter out empty string:
GET /_search
{
"query" : {
"regexp":{
"<your_field_name_here>": ".+"
}
}
}

Wrap a Missing Filter in the Must-Not section of a Bool Filter. It will only return documents where the field exists, and if you set the "null_value" property to true, values that are explicitly not null.
{
"query":{
"filtered":{
"query":{
"match_all":{}
},
"filter":{
"bool":{
"must":{},
"should":{},
"must_not":{
"missing":{
"field":"field1",
"existence":true,
"null_value":true
}
}
}
}
}
}
}

You can do that with bool query and combination of must and must_not like this:
GET index/_search
{
"query": {
"bool": {
"must": [
{"exists": {"field": "field1"}}
],
"must_not": [
{"term": {"field1": ""}}
]
}
}
}
I tested this with Elasticsearch 5.6.5 in Kibana.

The only solution here that worked for me in 5.6.5 was bigstone1998's regex answer. I'd prefer not to use a regex search though for performance reasons. I believe the reason the other solutions don't work is because a standard field will be analyzed and as a result have no empty string token to negate against. The exists query won't help on it's own either since an empty string is considered non-null.
If you can't change the index the regex approach may be your only option, but if you can change the index then adding a keyword subfield will solve the problem.
In the mappings for the index:
"myfield": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
}
Then you can simply use the query:
{
"query": {
"bool": {
"must": {
"exists": {
"field": "myfield"
}
},
"must_not": {
"term": {
"myfield.keyword": ""
}
}
}
}
}
Note the .keyword in the must_not component.

You can use not filter on top of missing.
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"not": {
"filter": {
"missing": {
"field": "searchField"
}
}
}
}
}
}

Here's the query example to check the existence of multiple fields:
{
"query": {
"bool": {
"filter": [
{
"exists": {
"field": "field_1"
}
},
{
"exists": {
"field": "field_2"
}
},
{
"exists": {
"field": "field_n"
}
}
]
}
}
}

You can use a bool combination query with must/must_not which gives great performance and returns all records where the field is not null and not empty.
bool must_not is like "NOT AND" which means field!="", bool must exist means its !=null.
so effectively enabling: where field1!=null and field1!=""
GET IndexName/IndexType/_search
{
"query": {
"bool": {
"must": [{
"bool": {
"must_not": [{
"term": { "YourFieldName": ""}
}]
}
}, {
"bool": {
"must": [{
"exists" : { "field" : "YourFieldName" }
}]
}
}]
}
}
}
ElasticSearch Version:
"version": {
"number": "5.6.10",
"lucene_version": "6.6.1"
}

ES 7.x
{
"_source": "field",
"query": {
"bool": {
"must": [
{
"exists": {
"field":"field"
}
}
],
"must_not": [
{
"term": {
"field.keyword": {
"value": ""
}
}
}
]
}
}
}

We are using Elasticsearch version 1.6 and I used this query from a co-worker to cover not null and not empty for a field:
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"bool": {
"must": [
{
"exists": {
"field": "myfieldName"
}
},
{
"not": {
"filter": {
"term": {
"myfieldName": ""
}
}
}
}
]
}
}
}
}
}

You need to use bool query with must/must_not and exists
To get where place is null
{
"query": {
"bool": {
"must_not": {
"exists": {
"field": "place"
}
}
}
}
}
To get where place is not null
{
"query": {
"bool": {
"must": {
"exists": {
"field": "place"
}
}
}
}
}

Elastic search Get all record where condition not empty.
const searchQuery = {
body: {
query: {
query_string: {
default_field: '*.*',
query: 'feildName: ?*',
},
},
},
index: 'IndexName'
};

Resources