How to write a conditional in a search query? - elasticsearch

I am searching among documents in a particular district. Documents have various statuses. The aim is to return all documents, except when document's status code is ABCD - such documents should only be returned if their ID is greater than 100. I have tried writing multiple queries, including the one below, which returns only the ABCD documents with ID greater than 100, and none of the other documents. What is wrong here? How can I get the non-ABCD documents as well?
"_source": true,
"from": 0,
"size": 50,
"sort": [
{
"firstStamp": "DESC"
}
],
"query": {
"bool": {
"must": [
{
"term": {
"districtId": "3755"
}
},
{
"bool": {
"must": [
{
"terms": {
"documentStatus.code.keyword": [
"ABCD"
]
}
},
{
"bool": {
"must": {
"script": {
"script": "doc['id'].value > 100"
}
}
}
}
]
}
}
]
}
}
}```

Since you have not added any index mapping, looking at your search
query data seems to be of object field data type. As far as I can
understand, your aim is to return all documents, except when the
document's status code is ABCD and document with status code ABCD
should only be returned if their ID is greater than 100.
Adding a working example with index data, search query, and search result
Index Data:
{
"id":200,
"documentStatus":{
"code":"DEF"
}
}
{
"id":200,
"documentStatus":{
"code":"ABCD"
}
}
{
"id":100,
"documentStatus":{
"code":"ABCD"
}
}
Search Query:
{
"query": {
"bool": {
"should": [
{
"bool": {
"must": [
{
"terms": {
"documentStatus.code.keyword": [
"ABCD"
]
}
},
{
"bool": {
"must": {
"script": {
"script": "doc['id'].value > 100"
}
}
}
}
]
}
},
{
"bool": {
"must_not": {
"terms": {
"documentStatus.code.keyword": [
"ABCD"
]
}
}
}
}
]
}
}
}
Search Result:
"hits": [
{
"_index": "stof_64351595",
"_type": "_doc",
"_id": "2",
"_score": 2.0,
"_source": {
"id": 200,
"documentStatus": {
"code": "ABCD"
}
}
},
{
"_index": "stof_64351595",
"_type": "_doc",
"_id": "3",
"_score": 0.0,
"_source": {
"id": 200,
"documentStatus": {
"code": "DEF"
}
}
}
]

You need to use must_not in your query if you want to have documents which don't have status code = ABCD. So your query would be some thing like this:
"from": 0,
"size": 50,
"sort": [
{
"firstStamp": "DESC"
}
],
{
"query": {
"bool": {
"must": [
{
"term": {
"districtId": "3755"
}
},
{
"range": {
"id": {
"gt": 100
}
}
}
],
"must_not": [
{
"terms": {
"documentStatus.code.keyword": [
"ABCD"
]
}
}
]
}
}
}

Related

Elastic Search Query on String Array Field

I'm working on Elastic Search and facing an issue regarding Array field. I've index named test-index with following mapping.
{
"test-index": {
"mappings": {
"properties": {
"courses": {
"type": "keyword"
}
}
}
}
}
My elastic search documents looks like this.
"hits": [
{
"_index": "test-index",
"_id": "1ac:0000000000_1",
"_score": 1,
"_source": {
"courses": [
"Course-1A",
"Course-1B",
"Course-1C",
"Course-1D",
"Course-1E",
"Course-1F"
]
}
},
{
"_index": "test-index",
"_id": "1ac:0000000000_2",
"_score": 1,
"_source": {
"courses": [
"Course-2A",
"Course-2B",
"Course-2C",
"Course-1A"
]
}
}
]
The document _id is my student ID. I want to get results with the maximum/highest relevance at the top and lowest on the bottom.
e.g
If I'm searching for courses ["Course-2A","Course-2B","Course-1C"] then user 1ac:0000000000_2 should appear at the top and user 1ac:0000000000_1 at the bottom.
I've tried following queries.
GET test-index/_search
{
"query": {
"bool": {
"must": [
{
"terms": {
"courses": [
"Course-1A",
"Course-2A",
"Course-2B"
]
}
}
]
}
}
}
User 1ac:0000000000_1 at the top and other at the bottom.
GET test-index/_search
{
"query": {
"bool": {
"should": [
{
"term": {
"courses": "Course-1A",
}
},
{
"term": {
"courses": "Course-2A",
}
},
{
"term": {
"courses": "Course-2B",
}
}
],
"minimum_should_match": "70%"
}
}
}
This gives me some desired results but not sure for larger dataset.

ElasticSearch: Aggregations for the count of documents that has empty nested array

Question- I want a count of documents where the nested array MATCHES is empty like "MATCHES": [ ].
My document structure looks like this(shows two records for simplicity) -
{
"hits": [
{
"_type": "_doc",
"_id": "ef0a2c44179a513476b080cc2a585d95",
"_source": {
"DIVISION_NUMBER": 44,
"MATCHES": [
{
"MATCH_STATUS": "APPROVED",
"UPDATED_ON": 1599171303000
}
]
}
},
{
"_type": "_doc",
"_id": "ef0a2c44179a513476b080cc2a585d95",
"_source": {
"DIVISION_NUMBER": 44,
"MATCHES": [ ]
}
}
]
}
Solution tried- I tried following different ways (workaround) of aggregation (empty-match-agg1,empty-match-agg2 ...) but none of these gave correct results. Please help!
"aggs": {
"sku": {
"nested": {
"path": "MATCHES"
},
"aggs": {
"empty-match-agg1": {
"missing": {
"field": "MATCHES"
}
},
"empty-match-agg2": {
"terms": {
"field": "MATCHES",
"missing": "N/A"
}
},
"empty-match-agg3": {
"sum": {
"script": {
"lang": "painless",
"source": "params['_source'].MATCHES"
}
}
},
"empty-match-agg4": {
"filter": {
"bool": {
"must_not": {
"nested": {
"query": {
"match_all": {}
},
"path": "MATCHES"
}
}
}
}
},
"empty-match-agg5": {
"terms": {
"field": "MATCHES"
}
}
}
}
}
Missing aggregation does not support nested field for now. There is open issue as of now.
To get count of empty matches, you can use a filter aggregation with the nested query wrapped into the must_not clause of the bool query.
{
"aggs": {
"missing_matches_agg": {
"filter": {
"bool": {
"must_not": {
"nested": {
"query": {
"match_all": {}
},
"path": "MATCHES"
}
}
}
}
}
}
}

ElasticSearch: Query nested array for empty and specific value in single query

Documents structure -
{
"hits": [
{
"_type": "_doc",
"_id": "ef0a2c44179a513476b080cc2a585d95",
"_source": {
"DIVISION_NUMBER": 44,
"MATCHES": [
{
"MATCH_STATUS": "APPROVED",
"UPDATED_ON": 1599171303000
}
]
}
},
{
"_type": "_doc",
"_id": "ef0a2c44179a513476b080cc2a585d95",
"_source": {
"DIVISION_NUMBER": 44,
"MATCHES": [ ]
}
}
]
}
Question - MATCHES is a nested array inside there is a text field MATCH_STATUS that can have any values say "APPROVED","REJECTED".
I am looking to search ALL documents that contain MATCH_STATUS having values say "APPROVED", "RECOMMENDED" as well as where there is no data in MATCHES (empty array "MATCHES": [ ]). Please note I want this in a single query.
I am able to do this in two separate queries like this -
GET all matches with status = RECOMMENDED, APPROVED
"must": [
{
"nested": {
"path": "MATCHES",
"query": {
"terms": {
"MATCHES.MATCH_STATUS.keyword": [
"APPROVED",
"RECOMMENDED"
]
}
}
}
}
]
GET all matches having empty array "MATCHES" : [ ]
{
"size": 5000,
"query": {
"bool": {
"filter": [],
"must_not": [
{
"nested": {
"path": "MATCHES",
"query": {
"exists": {
"field": "MATCHES"
}
}
}
}
]
}
},
"from": 0
}
You can combine both queries using should clause.
{
"query": {
"bool": {
"minimum_should_match": 1,
"should": [
{
"nested": {
"path": "MATCHES",
"query": {
"bool": {
"minimum_should_match": 1,
"should": [
{
"terms": {
"MATCHES.MATCH_STATUS.keyword": [
"APPROVED",
"RECOMMENDED"
]
}
}
]
}
}
}
},
{
"bool": {
"must_not": [
{
"nested": {
"path": "MATCHES",
"query": {
"bool": {
"filter": {
"exists": {
"field": "MATCHES"
}
}
}
}
}
}
]
}
}
]
}
}
}
Update: To answer your comment.
Missing aggregation does not support nested field for now. There is open issue as of now.
To get count of empty matches, you can use a filter aggregation with the nested query wrapped into the must_not clause of the bool query.
{
"aggs": {
"missing_matches_agg": {
"filter": {
"bool": {
"must_not": {
"nested": {
"query": {
"match_all": {}
},
"path": "MATCHES"
}
}
}
}
}
}
}

How to apply combination of filters on array of objects in elasticsearch?

Hi Please help me to write a suitable query for my task, I have list of products with different categories and associated attribute and tags. Here is two documents for two categories along with associated attribute list. There could be multiple attributes, just showing one.
{
"category": "blouses",
"attributes": [
{
"attribute": "women-blouse-neckline",
"tag": "round-neck"
}
]
}
{
"category": "dresses",
"attributes": [
{
"attribute": "women-dress-neckline",
"tag": "v-neck"
}
]
}
Now I want to get list of products from both categories which are dresses and blouses, but along with that a specific case at attribute level is :
Fetch all the products from dresses where attribute is women-dress-neckline and tag is v-neck along with all the products from blouses where attribute is women-blouse-neckline and tag is round-neck.
Thanks.
To query on each key of "attributes", you need to define "attributes" to be of the nested type, and then use a combination of bool/must and nested query
Adding a working example with index mapping, search query and search result
Index Mapping:
{
"mappings": {
"properties": {
"attributes": {
"type": "nested"
}
}
}
}
Search Query:
{
"query": {
"bool": {
"must": [
{
"match": {
"category": "dresses"
}
},
{
"nested": {
"path": "attributes",
"query": {
"bool": {
"must": [
{
"term": {
"attributes.attribute.keyword": "women-dress-neckline"
}
},
{
"term": {
"attributes.tag.keyword": "v-neck"
}
}
]
}
}
}
}
]
}
}
}
Search Result:
"hits": [
{
"_index": "69611494",
"_type": "_doc",
"_id": "2",
"_score": 2.0794413,
"_source": {
"category": "dresses",
"attributes": [
{
"attribute": "women-dress-neckline",
"tag": "v-neck"
}
]
}
}
]
Update 1:
You can combine two nested queries as shown below
{
"query": {
"bool": {
"should": [
{
"bool": {
"must": [
{
"match": {
"category": "dresses"
}
},
{
"nested": {
"path": "attributes",
"query": {
"bool": {
"must": [
{
"term": {
"attributes.attribute.keyword": "women-dress-neckline"
}
},
{
"term": {
"attributes.tag.keyword": "v-neck"
}
}
]
}
}
}
}
]
}
},
{
"bool": {
"must": [
{
"match": {
"category": "blouses"
}
},
{
"nested": {
"path": "attributes",
"query": {
"bool": {
"must": [
{
"term": {
"attributes.attribute.keyword": "women-blouse-neckline"
}
},
{
"term": {
"attributes.tag.keyword": "round-neck"
}
}
]
}
}
}
}
]
}
}
]
}
}
}

How to search on multiple fields of array in elasticsearch

I have a index in elastic search called professor
If for cross field i need "AND" condition
for same field array i need to OR condition
I need to search subject which is Physics or Accounting this is array of fields(OR) statement
AND
I need to search type is Permanent or GUEST condition this is array of fields(OR) statement
AND
I need to search Location is NY(&) condition
test = [{'id':1,'name': 'A','subject': ['Maths','Accounting'],'type':'Contract', 'Location':'NY'},
{ 'id':2,'name': 'AB','subject': ['Physics','Engineering'],'type':'Permanent','Location':'NY'},
{'id':3,'name': 'ABC','subject': ['Maths','Engineering'],'type':'Permanent','Location':'NY'},
{'id':4,'name':'ABCD','subject': ['Physics','Engineering'],'type':['Contract','Guest'],'Location':'NY'}]
Query is below,3rd one got it, How to add 1 and 2
content_search = es.search(index="professor", body={
"query": {
"bool": {
"must": {
"match_all": {}
},
"filter": [
{
"term": {
"Location.keyword": "NY"
}
}
]
}
}
})
content_search ['hits']['hits']
Expected out is id [{ 'id':2,'name': 'AB','subject': ['Physics','Engineering'],'type':'Permanent','Location':'NY'},{'id':4,'name':'ABCD','subject': ['Physics','Engineering'],'type':['Contract','Guest'],'Location':'NY'}]
The filter clause (query) must appear in matching documents. However
unlike must the score of the query will be ignored. Filter clauses are
executed in filter context, meaning that scoring is ignored and
clauses are considered for caching.
Please go through this Elasticsearch documentation on bool queries, to get a detailed understanding about it.
Adding a working example with index data(same as that in question), search query, and search result
Search Query:
{
"query": {
"bool": {
"must": {
"match": {
"Location.keyword": "NY"
}
},
"filter": [
{
"bool": {
"should": [
{
"match": {
"subject.keyword": "Accounting"
}
},
{
"match": {
"subject.keyword": "Physics"
}
}
]
}
},
{
"bool": {
"should": [
{
"match": {
"type.keyword": "Permanent"
}
},
{
"match": {
"type.keyword": "Guest"
}
}
]
}
}
]
}
}
}
Search Result:
"hits": [
{
"_index": "stof_64370980",
"_type": "_doc",
"_id": "2",
"_score": 0.10536051,
"_source": {
"id": 2,
"name": "AB",
"subject": [
"Physics",
"Engineering"
],
"type": "Permanent",
"Location": "NY"
}
},
{
"_index": "stof_64370980",
"_type": "_doc",
"_id": "4",
"_score": 0.10536051,
"_source": {
"id": 4,
"name": "ABCD",
"subject": [
"Physics",
"Engineering"
],
"type": [
"Contract",
"Guest"
],
"Location": "NY"
}
}
]
Another Search Query:
You can even use terms query that returns documents that contain
one or more exact terms in a provided field.The terms query is the
same as the term query, except you can search for multiple values.
{
"query": {
"bool": {
"must": [
{
"terms": {
"subject.keyword": [
"Physics",
"Accounting"
]
}
},
{
"terms": {
"type.keyword": [
"Guest",
"Permanent"
]
}
},
{
"match": {
"Location.keyword": "NY"
}
}
]
}
}
}
Update 1:
{
"query": {
"bool": {
"must": [
{
"terms": {
"subject.keyword": [
"Physics",
"Accounting"
]
}
},
{
"terms": {
"type.keyword": [
"Guest",
"Permanent"
]
}
},
{
"match": {
"Location.keyword": "NY"
}
},
{
"query_string": {
"query": "ABCD"
}
}
]
}
}
}

Resources