I have a following mapping applied to my index :
PUT /testing
PUT /testing/_mapping?pretty
{
"properties": {
"empID": {
"type":"long"
},
"state":{
"type":"text"
},
"Balance":{
"type":"long"
},
"loanid":{
"type":"long"
},
"rating":{
"type":"text"
},
"category":{
"type":"text"
}
}
}
Sample documents added in the index
POST testing/_doc?pretty
{
"empID":1,
"state":"NY",
"Balance":55,
"loanid":89,
"rating":"A",
"category":"PRO"
}
POST /testing/_doc?pretty
{
"empID":1,
"state":"TX",
"Balance":56,
"loanid":65,
"rating":"B",
"category":"TRIAL"
}
POST /testing/_doc?pretty
{
"empID":2,
"state":"TX",
"Balance":34,
"loanid":76,
"rating":"C",
"category":"PAID"
}
POST /testing/_doc?pretty
{
"empID":3,
"state":"TX",
"Balance":72,
"loanid":23,
"rating":"D",
"category":"FREE"
}
POST /testing/_doc?pretty
{
"dealID":3,
"state":"NY",
"Balance":23,
"loanid":67,
"rating":"E",
"category":"FREE"
}
POST /testing/_doc?pretty
{
"empID":2,
"state":"NY",
"Balance":23,
"loanid":98,
"rating":"F",
"category":"PRE"
}
POST /testing/_doc?pretty
{
"empID":2,
"state":"TX",
"Balance":19,
"loanid":100,
"rating":"D",
"category":"PAID"
}
I am trying to create ES query which is equivalent of sql query like :
select * from table_name
where empID =1 or state = 'NY'
and balance >=20 or loanid in (23, 67, 89) or rating = 'D'
and category!='FREE' or empID = 2 ;
vs (ES Query )
GET testing/_search?pretty
{
"query": {
"bool": {
"should": [
{
"match": {
"state": {
"query": "NY"
}
}
},
{
"term": {
"empID": 1
}
},
{
"bool": {
"must": [
{
"range": {
"Balance": {
"gte": 20
}
}
},
{
"bool": {
"should": [
{
"terms": {
"loanid": [
23,
67,
89
]
}
},
{
"match": {
"rating": {
"query": "D"
}
}
},
{
"bool": {
"must_not": [
{
"match": {
"category": {
"query": "FREE"
}
}
}
]
}
}
]
}
}
]
}
}
]
}
}
}
I am only getting 6 documents back wherein sql query gives 7 documents back .Could you confirm if this is how the multiple AND & OR QUERY would work in ES and help me in resolving the issue .
Related
GET test/_search
{
"query": {
"bool": {
"minimum_should_match": 1,
"should": [
{
"bool": {
"must": [
{
"term": {
"fragmentId": "1"
}
},
{
"term": {
"type": "fragment"
}
}
]
}
},
{
"bool": {
"must": [
{
"term": {
"fragmentId": "1"
}
},
{
"term": {
"type": "cf"
}
},
{
"range" :{
"start": {
"gte": 1,
"lte": 5
}
}
}
]
}
}
]
}
}
}
I am looking for two documents, one which has fragment id = 1, and type = fragment, whereas another where fragment id = 1, type = "cf" and start between 1 and 5.
The above query is doing the job, but I need to write type and fragment id twice. Is there a way I can add range condition only when the type is cf, basically clubbing both bools in one ?
This is the query you're looking for:
{
"query": {
"bool": {
"filter": [
{
"term": {
"fragmentId": "1"
}
}
],
"minimum_should_match": 1,
"should": [
{
"term": {
"type": "fragment"
}
},
{
"bool": {
"filter": [
{
"term": {
"type": "cf"
}
},
{
"range": {
"start": {
"gte": 1,
"lte": 5
}
}
}
]
}
}
]
}
}
}
I am trying to upgrade elastic-search to version 5. Previously I was using elastic-search version 2. I am having hard time converting OR query to bool[:should] query. Here is how my query looks like that was working in ES-2.
query: {:bool=>{
:should=>[
{:term=>{:user=>{:term=>70890}}},
{:term=>{:assignee=>{:term=>70890}}},
{:term=>{:participant=>{:term=>70890}}}],
:minimum_number_should_match=>1,
:filter=>[{:bool=> {:must_not=>{:exists=>{:field=>:date}}}},
{:term=>{:deleted=>false}},
{:or=>{:filters=>[
{:term=>{:user=>70890}},
{:term=>{:assignee=>70890}},
{:term=>{:private=>false}}
]}
}
]
}}
Query:
{
"bool": {
"should": [
{
"term": {
"user": {
"term": 70890
}
},
{
"term": {
"assignee": {
"term": 70890
}
}
},
{
"term": {
"participant": {
"term": 70890
}
}
}
],
"minimum_number_should_match": 1,
"filter": [
{
"bool": {
"must_not": {
"exists": {
"field": "date"
}
}
}
},
{
"term": {
"deleted": false
}
},
{
"or": {
"filters": [
{
"term": {
"user": 70890
}
},
{
"term": {
"assignee": 70890
}
},
{
"term": {
"private": false
}
}
]
}
}
]
}
}
I want to replace {:or=>{:filters}}. I have tried moving this part in :bool[:should] query but it gives wrong results.
q[:bool][:should] << {term: {user: 70890}}
q[:bool][:should] << {term: {assignee: 70890}}
q[:bool][:should] << {term: {private: false}}
q[:bool][:minimum_should_match] = 1
When I change minimum_should_match=2 it changes results. How do I fix it?
I am using following query to fetch all last 30 minutes records using elastic search, but I'm getting parsable error on line "now-30m".
Query:
{
"query": {
"bool": {
"must": [
{
"match": {
"appName": "magnus-alerting-system"
}
},
{
"match": {
"countryCode": "US2"
}
},
{
"match": {
"eventCode": 201
}
},
{
"match": {
"extractName": "LaborDemand"
}
},{
"range": {
"eventPostTimestamp": {
**"gte": "now()-30m"**
}
}
}
]
}
}
}
Error on Postman while executing service:
"root_cause": [
{
"type": "number_format_exception",
"reason": "For input string: \"now()-30m\""
}
]
PLease let me know how to correct it.
The reason is because now()-30m in elasticsearch is wrong since the correct format is just "now".
Documentation
Hence the correct query is the following:
{
"query": {
"bool": {
"must": [
{
"match": {
"appName": "magnus-alerting-system"
}
},
{
"match": {
"countryCode": "US2"
}
},
{
"match": {
"eventCode": 201
}
},
{
"match": {
"extractName": "LaborDemand"
}
},{
"range": {
"eventPostTimestamp": {
"gte": "now-30m"
}
}
}
]
}
}
}
The correct syntax for using data math in range query for date field would be as below:
{
"range": {
"eventPostTimestamp": {
"gte": "now-30m"
}
}
}
I have two queries in ES. Both have different turnaround time on the same set of documents. Both are doing the same thing conceptually. I have few doubts
1- What is the difference between these two?
2- Which one is better to use?
3- If both are same why they are performing differently?
1. Filtered bool
{
"from": 0,
"size": 5,
"query": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"term": {
"called_party_address_number": "1987112602"
}
},
{
"term": {
"original_sender_address_number": "6870340319"
}
},
{
"range": {
"x_event_timestamp": {
"gte": "2016-07-01T00:00:00.000Z",
"lte": "2016-07-30T00:00:00.000Z"
}
}
}
]
}
}
}
},
"sort": [
{
"x_event_timestamp": {
"order": "desc",
"ignore_unmapped": true
}
}
]
}
2. Simple Bool
{
"query": {
"bool": {
"must": [
{
"term": {
"called_party_address_number": "1277478699"
}
},
{
"term": {
"original_sender_address_number": "8020564722"
}
},
{
"term": {
"cause_code": "573"
}
},
{
"range": {
"x_event_timestamp": {
"gt": "2016-07-13T13:51:03.749Z",
"lt": "2016-07-16T13:51:03.749Z"
}
}
}
]
}
},
"from": 0,
"size": 10,
"sort": [
{
"x_event_timestamp": {
"order": "desc",
"ignore_unmapped": true
}
}
]
}
Mapping:
{
"ccp": {
"mappings": {
"type1": {
"properties": {
"original_sender_address_number": {
"type": "string"
},
"called_party_address_number": {
"type": "string"
},
"cause_code": {
"type": "string"
},
"x_event_timestamp": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
.
.
.
}
}
}
}
}
Update 1:
I tried bool/must query and bool/filter query on same set of data,but I found the strange behaviour
1-
bool/must query is able to search the desired document
{
"query": {
"bool": {
"must": [
{
"term": {
"called_party_address_number": "8701662243"
}
},
{
"term": {
"cause_code": "401"
}
}
]
}
}
}
2-
While bool/filter is not able to search the document. If I remove the second field condition it searches the same record with field2's value as 401.
{
"query": {
"bool": {
"filter": [
{
"term": {
"called_party_address_number": "8701662243"
}
},
{
"term": {
"cause_code": "401"
}
}
]
}
}
}
Update2:
Found a solution of suppressing scoring phase with bool/must query by wrapping it within "constant_score".
{
"query": {
"constant_score": {
"filter": {
"bool": {
"must": [
{
"term": {
"called_party_address_number": "1235235757"
}
},
{
"term": {
"cause_code": "304"
}
}
]
}
}
}
}
}
Record we are trying to match have "called_party_address_number": "1235235757" and "cause_code": "304".
The first one uses the old 1.x query/filter syntax (i.e. filtered queries have been deprecated in favor of bool/filter).
The second one uses the new 2.x syntax but not in a filter context (i.e. you're using bool/must instead of bool/filter). The query with 2.x syntax which is equivalent to your first query (i.e. which runs in a filter context without score calculation = faster) would be this one:
{
"query": {
"bool": {
"filter": [
{
"term": {
"called_party_address_number": "1277478699"
}
},
{
"term": {
"original_sender_address_number": "8020564722"
}
},
{
"term": {
"cause_code": "573"
}
},
{
"range": {
"x_event_timestamp": {
"gt": "2016-07-13T13:51:03.749Z",
"lt": "2016-07-16T13:51:03.749Z"
}
}
}
]
}
},
"from": 0,
"size": 10,
"sort": [
{
"x_event_timestamp": {
"order": "desc",
"ignore_unmapped": true
}
}
]
}
ElasticSearch Version: 0.90.1, JVM: 1.6.0_51(20.51-b01-457)
I'm trying to do two things with my ElasticSearch query: 1) filter the results based on a boolean (searchable) and "open_date < tomorrow" and 2) two sort by the field "open_date" DESC
This produces the following query:
{
"query": {
"bool": {
"should": [
{
"prefix": {
"name": "foobar"
}
},
{
"query_string": {
"query": "foobar"
}
},
{
"match": {
"name": {
"query": "foobar"
}
}
}
],
"minimum_number_should_match": 1
},
"filtered": {
"filter": {
"and": [
{
"term": {
"searchable": true
}
},
{
"range": {
"open_date": {
"lt": "2013-07-16"
}
}
}
]
}
}
},
"sort": [
{
"open_date": "desc"
}
]
}
However, the results that come back are not being sorted by "open_date". If I remove the filter:
{
"query": {
"bool": {
"should": [
{
"prefix": {
"name": "foobar"
}
},
{
"query_string": {
"query": "foobar"
}
},
{
"match": {
"name": {
"query": "foobar"
}
}
}
],
"minimum_number_should_match": 1
}
},
"sort": [
{
"open_date": "desc"
}
]
}
... the results come back as expected.
Any ideas?
I'm not sure about the Tire code, but the JSON does not correctly construct a filtered query. My guess is that this overflows and causes the sort element to also not be correctly parsed.
A filtered query should be constructed like this (see http://www.elasticsearch.org/guide/reference/query-dsl/filtered-query/ ):
{
"query": {
"filtered": { // Note: this contains both query and filter
"query": {
"bool": {
"should": [
{
"prefix": {
"name": "foobar"
}
},
{
"query_string": {
"query": "foobar"
}
},
{
"match": {
"name": {
"query": "foobar"
}
}
}
],
"minimum_number_should_match": 1
}
},
"filter": {
"and": [
{
"term": {
"searchable": true
}
},
{
"range": {
"open_date": {
"lt": "2013-07-16"
}
}
}
]
}
}
},
"sort": [
{
"open_date": "desc"
}
]
}
Cheers,
Boaz