Must not query elasticsearch - elasticsearch

I have my request:
{
"size": 10,
"query": {
"filtered": {
"query": {
"bool": {
"must": [
{"term": {"event": "matchmaking_done"}}
]
}
},
"filter": {
"range": {
"#timestamp": {
"gt" : "2016-06-01T00:00:00.000Z",
"lte" : "2016-06-01T00:05:00.000Z"
}
}
}
}
},
"aggs" : {
"user-ids" : {
"terms" : { "field" : "user_id",
"size": 0
}
}
}
}
And I need to add into this request parameter - does not contain field pvp_league! I tried add must_not but can't understand how to do this correct.
Help please!

You answered it yourself, but the ES 2.x way to do this is to not use the filtered query because it has been deprecated and it will be removed in ES 5.0. ES 2.x introduces the concept of the "filter" context rather than every query being either just a query or a filter; now every query is both a filter or a query (scored), just depending on the context it's used in.
For your query, this therefore becomes a little simpler because of the simplified bool / filter syntax:
{
"size":10,
"query":{
"bool":{
"must":[
{
"term":{
"event":"matchmaking_done"
}
}
],
"must_not":[
{
"exists":{
"field":"pvp_league"
}
}
],
"filter":[
{
"range":{
"#timestamp":{
"gt":"2016-06-01T00:00:00.000Z",
"lte":"2016-06-01T00:05:00.000Z"
}
}
}
]
}
},
"aggs":{
"user-ids":{
"terms":{
"field":"user_id",
"size":0
}
}
}
}
As a very big aside, specifying "size" : 0 for the terms aggregation, you are requesting all unique terms, up to INT_MAX. That is not a scalable request (works great with 10 user_ids, or even 100, but not 10000 users).
As a not-so-bad aside, your request doesn't need a query context at all because nothing about the search side of it cares about relevance. Your term query ("event" : "matchmaking_done") either matches or it doesn't. Since you either want it to match or not, but you don't really care about order inherently, you should use this in the filter context. This changes the request to:
{
"size": 10,
"query": {
"bool": {
"must_not": [
{
"exists": {
"field": "pvp_league"
}
}
],
"filter":[
{
"range": {
"#timestamp": {
"gt":"2016-06-01T00:00:00.000Z",
"lte":"2016-06-01T00:05:00.000Z"
}
}
},
{
"term": {
"event": "matchmaking_done"
}
}
]
}
},
"aggs": {
"user-ids": {
"terms": {
"field": "user_id",
"size": 0
}
}
}
}

I've found solution! It looks like this:
{
"size": 10,
"query": {
"filtered": {
"query": {
"bool": {
"must": [
{"term": {"event": "matchmaking_done"}}
],
"must_not": [
{"filtered": {
"filter": {
"exists": {
"field": "pvp_league"
}
}
}
}
]
}
},
"filter": {
"range": {
"#timestamp": {
"gt" : "2016-06-01T00:00:00.000Z",
"lte" : "2016-06-01T00:05:00.000Z"
}
}
}
}
},
"aggs" : {
"user-ids" : {
"terms" : { "field" : "user_id",
"size": 0
}
}
}
}

Related

nested boolean with match query in elasticsearch

I would like to match within a boolean query in Elasticsearch. I have the match query and boolean query working as expected now, but I am not sure how to have a AND to combine them.
nested boolean
{
"query": {
"constant_score" : {
"filter":{
"bool":{
"must":[
{"terms":{"address.keyword": addr}},
{"bool":{
"should":[
{"terms": {"state.keyword": state}}
,{"terms": {"city.keyword": city}}
]
}}
]
}
}
}}}
match
{"query": {
"match": {
"auct_title": {
"query": keyword,
"operator": "and"
}
}
}
, "collapse" : {
"field" : "id"
}
,"sort" : [
{ sort_field: {"order" : sort_order} }]
,"size":20
}
You can move natch to the must clause . So document has to satisfy three conditions
1.address
2.either of state/city
2.match on auct_title
It will then return one document per Id based on sort order passed
GET <index>/_search
{
"query": {
"constant_score": {
"filter": {
"bool": {
"must": [
{
"term": {
"address.keyword": "addr"
}
},
{
"bool": {
"should": [
{
"term": {
"state.keyword": "state"
}
},
{
"term": {
"city.keyword": "city"
}
}
]
}
},
{
"match": {
"auct_title": {
"query": "keyword",
"operator": "and"
}
}
}
]
}
}
}
},
"collapse": {
"field": "id"
},
"sort": [
{
"FIELD": {
"order": "desc"
}
}
],
"size": 20
}

Aggregation, Query Context and filter Context not working in Elasticsearch 5.1

I am facing issue in migrating from elastic search 1.5 to 5.1.
Following is my elastic search - 1.5 Query:
{
"_source":["_id","spotlight"],
"query":{
"filtered":{
"filter":{
"and":[
{"term":{"gender":"female"}},
{"range":{"lastlogindate":{"gte":"2016-10-19 12:39:57"}}}
]
}
}
},
"filter":{
"and":[
{"term":{"maritalstatus":"1"}}
]
},
"sort":[{"member2_dummy7":{"order":"desc"}}],
"size":"0",
"aggs": {
"maritalstatus": {
"filter": {},
"aggs" : {
"filtered_maritalstatus": {"terms":{"field":"maritalstatus","size":5000}}
}
}
}
}
This query is giving me correct doc_count in aggregations. This doc_count is calculated over result set returned by query context and it ignores filter context.
I have written same query in elastic search 5.1:
{
"_source":["_id","spotlight"],
"query":{
"bool":{
"must":[
{"term":{"gender":"female"}},
{"range":{"lastlogindate":{"gte":"2016-10-19 12:39:57"}}}
],
"filter":{
"bool":{
"must":[
{"term":{"maritalstatus":"1"}}
]
}
}
}
},
"sort":[{"member2_dummy7":{"order":"DESC"}}],
"size":"0",
"aggs": {
"maritalstatus": {
"filter": {},
"aggs" : {
"filtered_maritalstatus": {"terms":{"field":"maritalstatus","size":5000}}
}
}
}
}
But in elastic search 5.1, it is returning me wrong doc_count in aggregation. I think it is taking filter in query context and hence, it is returning wrong doc_cout. Can someone tell me correct way to separate query and filter in elastic search 5.1?
Your 1.5 query uses post_filter which you have removed in your 5.1 query.
The equivalent query in ES 5.1 is the following (filtered/filter simply gets replaced as bool/filter and the top-level filter renamed to post_filter):
{
"_source": [
"_id",
"spotlight"
],
"query": {
"bool": {
"filter": [
{
"term": {
"gender": "female"
}
},
{
"range": {
"lastlogindate": {
"gte": "2016-10-19 12:39:57"
}
}
}
]
}
},
"post_filter": {
"term": {
"maritalstatus": "1"
}
},
"sort": [
{
"member2_dummy7": {
"order": "desc"
}
}
],
"size": "0",
"aggs": {
"maritalstatus": {
"filter": {},
"aggs": {
"filtered_maritalstatus": {
"terms": {
"field": "maritalstatus",
"size": 5000
}
}
}
}
}
}

ElasticSearch date range

I have the following query:
{
"query": {
"query_string": {
"query": "searchTerm",
"default_operator": "AND"
}
},
"facets": {
"counts": {
"date_histogram": {
"field": "firstdate",
"interval": "hour"
}
}
}
and I would like to add a date range to it, so as to retrieve values for the field firstdate which are within a specific from/to interval. Any suggestions on how to do it? Many thanks!
you just need to add a range filter to your query:
{
"query":{
"filtered": {
"query": {
"query_string": {"query": "searchTerm", "default_operator": "AND" }
},
"filter" : {
"range": {"firstdate": {"gte": "2014-10-21T20:03:12.963","lte": "2014-11-24T20:03:12.963"}}
}
}
},
"facets": {
"counts": {
"date_histogram": {
"field": "firstdate",
"interval": "hour"
}
}
}
}
Boolean query will work too,
{
"query" :{
"bool" : {
"must" : {
"range": {"firstdate": {"gte": "2014-10-21T20:03:12.963","lte": "2014-11-24T20:03:12.963"}}
},
"must" : {
"query_string": {
"query": "searchTerm",
"default_operator": "AND"
}
}
}
},
"facets": {
"counts": {
"date_histogram": {
"field": "firstdate",
"interval": "hour"
}
}
}
}
This query displays the results which appears in the given date range. "date_field_name" is the field name on which you want to set date range filters.
GET index_name/_search
{
"query": {
"bool": {
"must":[
{
"range": {
"date_field_name": {
"gte": "2019-09-23 18:30:00",
"lte": "2019-09-24 18:30:00"
}
}
}
]
}
},
"size": 10
}
https://your_elasticsearch/your_index PUT
{
"mappings": {
"properties": {
"created_at": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss"
}
}
}
}
https://your_elasticsearch/your_index/_search POST
{
"query": {
"bool": {
"filter": [
{
"range": {
"created_at": {
"gte": "2020-04-01 08:03:12",
"lte": "2020-04-01 20:03:12"
}
}
}
]
}
}
}

Multiple filters and an aggregate in elasticsearch

How can I use a filter in connection with an aggregate in elasticsearch?
The official documentation gives only trivial examples for filter and for aggregations and no formal description of the query dsl - compare it e.g. with postgres documentation.
Through trying out I found following query, which is accepted by elasticsearch (no parsing errors), but ignores the given filters:
{
"filter": {
"and": [
{
"term": {
"_type": "logs"
}
},
{
"term": {
"dc": "eu-west-12"
}
},
{
"term": {
"status": "204"
}
},
{
"range": {
"#timestamp": {
"from": 1398169707,
"to": 1400761707
}
}
}
]
},
"size": 0,
"aggs": {
"time_histo": {
"date_histogram": {
"field": "#timestamp",
"interval": "1h"
},
"aggs": {
"name": {
"percentiles": {
"field": "upstream_response_time",
"percents": [
98.0
]
}
}
}
}
}
}
Some people suggest using query instead of filter. But the official documentation generally recommends the opposite for filtering on exact values. Another issue with query: while filters offer an and, query does not.
Can somebody point me to documentation, a blog or a book, which describe writing non-trivial queries: at least an aggregate plus multiple filters.
I ended up using a filter aggregation - not filtered query. So now I have 3 nested aggs elements.
I also use bool filter instead of and as recommended by #alex-brasetvik because of http://www.elasticsearch.org/blog/all-about-elasticsearch-filter-bitsets/
My final implementation:
{
"aggs": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"term": {
"_type": "logs"
}
},
{
"term": {
"dc": "eu-west-12"
}
},
{
"term": {
"status": "204"
}
},
{
"range": {
"#timestamp": {
"from": 1398176502000,
"to": 1400768502000
}
}
}
]
}
},
"aggs": {
"time_histo": {
"date_histogram": {
"field": "#timestamp",
"interval": "1h"
},
"aggs": {
"name": {
"percentiles": {
"field": "upstream_response_time",
"percents": [
98.0
]
}
}
}
}
}
}
},
"size": 0
}
Put your filter in a filtered-query.
The top-level filter is for filtering search hits only, and not facets/aggregations. It was renamed to post_filter in 1.0 due to this quite common confusion.
Also, you might want to look into this post on why you often want to use bool and not and/or: http://www.elasticsearch.org/blog/all-about-elasticsearch-filter-bitsets/
more on #geekQ 's answer: to support filter string with space char,for multipal term search,use below:
{ "aggs": {
"aggresults": {
"filter": {
"bool": {
"must": [
{
"match_phrase": {
"term_1": "some text with space 1"
}
},
{
"match_phrase": {
"term_2": "some text with also space 2"
}
}
]
}
},
"aggs" : {
"all_term_3s" : {
"terms" : {
"field":"term_3.keyword",
"size" : 10000,
"order" : {
"_term" : "asc"
}
}
}
}
} }, "size": 0 }
Just for reference, as for the version 7.2, I tried with something as follows to achieve multiple filters for aggregation:
filter aggregation to filter for aggregation
use bool to set up the compound query
POST movies/_search?size=0
{
"size": 0,
"aggs": {
"test": {
"filter": {
"bool": {
"must": {
"term": {
"genre": "action"
}
},
"filter": {
"range": {
"year": {
"gte": 1800,
"lte": 3000
}
}
}
}
},
"aggs": {
"year_hist": {
"histogram": {
"field": "year",
"interval": 50
}
}
}
}
}
}

Create Elasticsearch curl query for not null and not empty("")

How can i create Elasticsearch curl query to get the field value which are not null and not empty(""),
Here is the mysql query:
select field1 from mytable where field1!=null and field1!="";
A null value and an empty string both result in no value being indexed, in which case you can use the exists filter
curl -XGET 'http://127.0.0.1:9200/test/test/_search?pretty=1' -d '
{
"query" : {
"constant_score" : {
"filter" : {
"exists" : {
"field" : "myfield"
}
}
}
}
}
'
Or in combination with (eg) a full text search on the title field:
curl -XGET 'http://127.0.0.1:9200/test/test/_search?pretty=1' -d '
{
"query" : {
"filtered" : {
"filter" : {
"exists" : {
"field" : "myfield"
}
},
"query" : {
"match" : {
"title" : "search keywords"
}
}
}
}
}
'
As #luqmaan pointed out in the comments, the documentation says that the filter exists doesn't filter out empty strings as they are considered non-null values.
So adding to #DrTech's answer, to effectively filter null and empty string values out, you should use something like this:
{
"query" : {
"constant_score" : {
"filter" : {
"bool": {
"must": {"exists": {"field": "<your_field_name_here>"}},
"must_not": {"term": {"<your_field_name_here>": ""}}
}
}
}
}
}
On elasticsearch 5.6, I have to use command below to filter out empty string:
GET /_search
{
"query" : {
"regexp":{
"<your_field_name_here>": ".+"
}
}
}
Wrap a Missing Filter in the Must-Not section of a Bool Filter. It will only return documents where the field exists, and if you set the "null_value" property to true, values that are explicitly not null.
{
"query":{
"filtered":{
"query":{
"match_all":{}
},
"filter":{
"bool":{
"must":{},
"should":{},
"must_not":{
"missing":{
"field":"field1",
"existence":true,
"null_value":true
}
}
}
}
}
}
}
You can do that with bool query and combination of must and must_not like this:
GET index/_search
{
"query": {
"bool": {
"must": [
{"exists": {"field": "field1"}}
],
"must_not": [
{"term": {"field1": ""}}
]
}
}
}
I tested this with Elasticsearch 5.6.5 in Kibana.
The only solution here that worked for me in 5.6.5 was bigstone1998's regex answer. I'd prefer not to use a regex search though for performance reasons. I believe the reason the other solutions don't work is because a standard field will be analyzed and as a result have no empty string token to negate against. The exists query won't help on it's own either since an empty string is considered non-null.
If you can't change the index the regex approach may be your only option, but if you can change the index then adding a keyword subfield will solve the problem.
In the mappings for the index:
"myfield": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
}
Then you can simply use the query:
{
"query": {
"bool": {
"must": {
"exists": {
"field": "myfield"
}
},
"must_not": {
"term": {
"myfield.keyword": ""
}
}
}
}
}
Note the .keyword in the must_not component.
You can use not filter on top of missing.
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"not": {
"filter": {
"missing": {
"field": "searchField"
}
}
}
}
}
}
Here's the query example to check the existence of multiple fields:
{
"query": {
"bool": {
"filter": [
{
"exists": {
"field": "field_1"
}
},
{
"exists": {
"field": "field_2"
}
},
{
"exists": {
"field": "field_n"
}
}
]
}
}
}
You can use a bool combination query with must/must_not which gives great performance and returns all records where the field is not null and not empty.
bool must_not is like "NOT AND" which means field!="", bool must exist means its !=null.
so effectively enabling: where field1!=null and field1!=""
GET IndexName/IndexType/_search
{
"query": {
"bool": {
"must": [{
"bool": {
"must_not": [{
"term": { "YourFieldName": ""}
}]
}
}, {
"bool": {
"must": [{
"exists" : { "field" : "YourFieldName" }
}]
}
}]
}
}
}
ElasticSearch Version:
"version": {
"number": "5.6.10",
"lucene_version": "6.6.1"
}
ES 7.x
{
"_source": "field",
"query": {
"bool": {
"must": [
{
"exists": {
"field":"field"
}
}
],
"must_not": [
{
"term": {
"field.keyword": {
"value": ""
}
}
}
]
}
}
}
We are using Elasticsearch version 1.6 and I used this query from a co-worker to cover not null and not empty for a field:
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"bool": {
"must": [
{
"exists": {
"field": "myfieldName"
}
},
{
"not": {
"filter": {
"term": {
"myfieldName": ""
}
}
}
}
]
}
}
}
}
}
You need to use bool query with must/must_not and exists
To get where place is null
{
"query": {
"bool": {
"must_not": {
"exists": {
"field": "place"
}
}
}
}
}
To get where place is not null
{
"query": {
"bool": {
"must": {
"exists": {
"field": "place"
}
}
}
}
}
Elastic search Get all record where condition not empty.
const searchQuery = {
body: {
query: {
query_string: {
default_field: '*.*',
query: 'feildName: ?*',
},
},
},
index: 'IndexName'
};

Resources