ElasticSearch multiple AND/OR query - elasticsearch

Question

I have a schema like below -
{
"errorCode": "e015",
"errorDescription": "Description e015",
"storeId": "71102",
"businessFunction": "PriceFeedIntegration",
"createdDate": "2021-02-20T09:17:04.004",
"readBy": [
{
"userId": "scha3055"
},
{
"userId": "abcd1234"
}
]
}
I'm trying to search combination of "errorCode","storeId","businessFunction" with a date range like below -
{
"query": {
"bool": {
"must": [
{
"terms": {
"errorCode": [
"e015",
"e020",
"e022"
]
}
},
{
"terms": {
"storeId": [
"71102",
"71103"
]
}
},
{
"range": {
"createdDate": {
"gte": "2021-02-16T09:17:04.000",
"lte": "2021-02-22T00:00:00.005"
}
}
}
]
}
}
}
But when I add another condition with "businessFunction" the query does not work.
{
"query": {
"bool": {
"must": [
{
"terms": {
"errorCode": [
"e015",
"e020",
"e022"
]
}
},
{
"terms": {
"storeId": [
"71102",
"71103"
]
}
},
{
"terms": {
"errorDescription": [
"Description e020",
"71103"
]
}
},
{
"range": {
"createdDate": {
"gte": "2021-02-16T09:17:04.000",
"lte": "2021-02-22T00:00:00.005"
}
}
}
]
}
}
}
What am I missing in the query? When I add the third "terms" cndition , the query does not work. Please suggest or let me know any alternate way.

Answer 1

In your example you are searching for "Description e020" but in your example you stored "Description e015".
Short answer, I hope that's right for you:
"Description e015" will have been indexed as the two terms ["description","e015"].
use match_phrase instead of terms
...
{
"match_phrase": {
"errorDescription": "Description e015"
}
},
{
"range": {
"createdDate": {
"gte": "2021-02-16T09:17:04.000",
"lte": "2021-02-22T00:00:00.005"
}
}
}
....
Without knowing your mapping, I think that your errorDescription field its analyzed.
Other option not recommended:
If your field its analyzed and you require match exact, search in errorDescription.keyword
{
"terms": {
"errorDescription.keyword": [
"Description e015"
]
}
}
UPDATE
Long answer:
As I mentioned previously maybe, your field value was analyzed, then converted from "PriceFeedIntegration2" to pricefeedintegration2.
2 options
Search by your field.keyword aka businessFunction.keyword
Change your field mapping to not analyzed. Then you can get results just as you expect using terms.
Option: 1
It's the easy way, if you never run full text searches on that field, better not use as default. If it does not matter, use this option, it is the simplest.
Check your businessFunction.keyword field (created by default if you dont specify mapping)
Indexing data without mapping on my000001 index
curl -X "POST" "http://localhost:9200/my000001/_doc" \
-H "Content-type: application/json" \
-d $'
{
"errorCode": "e015",
"errorDescription": "Description e015",
"storeId": "71102",
"businessFunction": "PriceFeedIntegration",
"createdDate": "2021-02-20T09:17:04.004"
}'
Check
curl -X "GET" "localhost:9200/my000001/_analyze" \
-H "Content-type: application/json" \
-d $'{
"field": "businessFunction.keyword",
"text": "PriceFeedIntegration"
}'
Result:
{
"tokens": [
{
"token": "PriceFeedIntegration",
"start_offset": 0,
"end_offset": 20,
"type": "word",
"position": 0
}
]
}
Get the results using businessFunction.keyword
curl -X "GET" "localhost:9200/my000001/_search" \
-H "Content-type: application/json" \
-d $'{
"query": {
"bool": {
"must": [
{
"terms": {
"errorCode": [
"e015",
"e020",
"e022"
]
}
},
{
"terms": {
"storeId": [
"71102",
"71103"
]
}
},
{
"terms": {
"businessFunction.keyword": [
"PriceFeedIntegration2",
"PriceFeedIntegration"
]
}
},
{
"range": {
"createdDate": {
"gte": "2021-02-16T09:17:04.000",
"lte": "2021-02-22T00:00:00.005"
}
}
}
]
}
}
}' | jq
Why isn't recommended as default option?
"The default dynamic string mappings will index string fields both as
text and keyword. This is wasteful if you only need one of them."
https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-disk-usage.html
Option 2
Run on my000001 index
curl -X "GET" "localhost:9200/my000001/_analyze" \
-H "Content-type: application/json" \
-d $'{
"field": "businessFunction",
"text": "PriceFeedIntegration"
}'
You can see, that your field value was analyzed(tokenized, lowercase, and others modifications depending of the analyzer and the value provided)
Results:
{
"tokens": [
{
"token": "pricefeedintegration",
"start_offset": 0,
"end_offset": 20,
"type": "<ALPHANUM>",
"position": 0
}
]
}
That is the reason why your search doesn't return results.
"PriceFeedIntegration" doesn't match with "pricefeedintegration"
"The problem isn’t with the term query; it is with the way the data
has been indexed."
Your businessFunction field value was analyzed.
If you require find(search/filter) by exact values, maybe you need to change your "businessFunction" field mapping to not_analyzed.
Change your mapping require delete your index and create again providing the required mapping.
If you try to change the mapping of an existing index you will get an "resource_already_exists_exception" error.
Here is the background that you need to know in order to solve your problem:
https://www.elastic.co/guide/en/elasticsearch/guide/master/_finding_exact_values.html#_finding_exact_values
Create a Mapping on a new my000005 index
curl -X "PUT" "localhost:9200/my000005" \
-H "Content-type: application/json" \
-d $'{
"mappings" : {
"properties" : {
"businessFunction" : {
"type" : "keyword"
},
"errorDescription" : {
"type" : "text"
},
"errorCode" : {
"type" : "keyword"
},
"createdDate" : {
"type" : "date"
},
"storeId": {
"type" : "keyword"
}
}
}
}'
Indexing data
curl -X "POST" "http://localhost:9200/my000005/_doc" \
-H "Content-type: application/json" \
-d $'
{
"errorCode": "e015",
"errorDescription": "Description e015",
"storeId": "71102",
"businessFunction": "PriceFeedIntegration",
"createdDate": "2021-02-20T09:17:04.004"
}'
Get the results, that you expect using terms businessFunction
curl -X "GET" "localhost:9200/my000005/_search" \
-H "Content-type: application/json" \
-d $'{
"query": {
"bool": {
"must": [
{
"terms": {
"errorCode": [
"e015",
"e020",
"e022"
]
}
},
{
"terms": {
"storeId": [
"71102",
"71103"
]
}
},
{
"terms": {
"businessFunction": [
"PriceFeedIntegration2",
"PriceFeedIntegration"
]
}
},
{
"range": {
"createdDate": {
"gte": "2021-02-16T09:17:04.000",
"lte": "2021-02-22T00:00:00.005"
}
}
}
]
}
}
}' | jq
This answer is based on what I think is your mapping and your needs.
In the future share your mapping and your ES version, in order to get a better answer from the community.
curl -X "GET" "localhost:9200/yourindex/_mappings"
Please read this https://www.elastic.co/guide/en/elasticsearch/guide/master/_finding_exact_values.html#_finding_exact_values
and this https://www.elastic.co/blog/strings-are-dead-long-live-strings