Nested aggregation in nested aggregation query - elasticsearch

Having the below (abbreviated) document in Elastic Search 7.1. Focusing on questions.influencerReponse.selectAllThatApplyResponses path.
{
"questions": [
{
"questionId": "79cfc6e7-731e-4d83-9dd6-82f4f39fff03",
"questionKind": "select_all_that_apply",
"questionText": "Have you heard of any of the following charities?",
"questionOptions": {
"1": "Plan International",
"2": "Young Women's Trust",
"3": "Women For Refugee Women",
"4": "The FPA"
},
"influencerReponse": {
"questionId": "79cfc6e7-731e-4d83-9dd6-82f4f39fff03",
"questionKind": "select_all_that_apply",
"text": null,
"questionOrder": 3,
"order": null,
"shortAnswerResponse": null,
"viewerSentimentResponse": null,
"yesNoResponse": null,
"selectAllThatApplyResponses": [
{
"key": "2",
"value": "Young Women's Trust"
}
]
}
}
]
}
I want to get the term aggregations for the key or the value, both are keyword type. I accomplished that before but not in the level of selectAllThatApplyResponses nested type.
Here's what I have so far and throwing the below error.
{
"query": {
"bool": {
"must": [
{
"term": {
"sponsorshipId": {
"value": "33c7140f-23ae-46f2-a0fe-49e2251114e4"
}
}
}
]
}
},
"track_total_hits": true,
"size": 0,
"aggs": {
"select_all_that_apply_responses": {
"nested": {
"path": "questions"
},
"aggs": {
"filter_types": {
"filter": {
"bool": {
"must": [
{
"match": {
"questions.questionId": "79cfc6e7-731e-4d83-9dd6-82f4f39fff03"
}
}
]
}
},
"aggs": {
"select_all_that_apply_nested": {
"nested": {
"path": "questions.influencerReponse.selectAllThatApplyResponses"
},
"aggs": {
"terms": {
"field": "questions.influencerReponse.selectAllThatApplyResponses.key"
}
}
}
}
}
}
}
}
}
I am receiving the below error.
{
"error": {
"root_cause": [
{
"type": "parsing_exception",
"reason": "Expected [START_OBJECT] under [field], but got a [VALUE_STRING] in [terms]",
"line": 42,
"col": 46
}
],
"type": "parsing_exception",
"reason": "Expected [START_OBJECT] under [field], but got a [VALUE_STRING] in [terms]",
"line": 42,
"col": 46
},
"status": 400
}

The final terms agg needs a name too -- I called it select_all_that_apply_nested_terms.
...
"select_all_that_apply_nested":{
"nested":{
"path":"questions.influencerReponse.selectAllThatApplyResponses"
},
"aggs":{
"select_all_that_apply_nested_terms":{
"terms":{
"field":"questions.influencerReponse.selectAllThatApplyResponses.key"
}
}
}
}
...

Related

Elastic search match query with pattern

I have a column where all the values start with "ARK". For example
number
ARK101223
ARK123422
ARK234002
ARK234177
I need to get all the records for the column number that matches with ARK using elastic search. Whereever I have the number as column and matches with ARK, I need to retrieve those records only. Some records will not have number as column so I want those to be ignored..
Below is the query that I tried but not working
{
"query": {
"bool": {
"must": [
{
"prefix": {
"number.keyword": "ARK"
}
},
{
"range": {
"date_1": {
"gte": "2022-01-01 01:00:00",
"lte": "2022-03-10 01:00:00"
}
},
"sort": [
{
"date_1": {
"order": "asc"
},
"date_2": {
"order": "asc"
},
"ts": {
"order": "asc"
}
}
]
}
]
}
}
}
Below is the error:
{
"error": {
"root_cause": [
{
"type": "parsing_exception",
"reason": "[range] malformed query, expected [END_OBJECT] but found [FIELD_NAME]",
"line": 1,
"col": 155
}
],
"type": "x_content_parse_exception",
"reason": "[1:155] [bool] failed to parse field [must]",
"caused_by": {
"type": "parsing_exception",
"reason": "[range] malformed query, expected [END_OBJECT] but found [FIELD_NAME]",
"line": 1,
"col": 155
}
},
"status": 400
}
If Elasticsearch generated mapping for your index, than you will have .keyword field for your number text field and on that you can make prefix query to get expected result.
{
"query": {
"prefix": {
"number.keyword": "ARK"
}
}
}
Update:
{
"query": {
"bool": {
"must": [
{
"prefix": {
"number.keyword": "ARK"
}
},
{
"range": {
"date_1": {
"gte": "2022-01-01 01:00:00",
"lte": "2022-03-10 01:00:00"
}
}
}
]
}
},
"sort": [
{
"date_1": {
"order": "asc"
},
"date_2": {
"order": "asc"
},
"ts": {
"order": "asc"
}
}
]
}

ES plugin query

I am running the following query and getting an error:
Query :
POST /sbl_nmon2019.12.02/_search?size=0
{"query":{
"bool":{
"must" : [{
"range":{"#timestamp":{"gte": "now-30m"}},
"aggs":{"max_cpu" : {"field":"cpu_consumed"}},
"match":{"Server" : "siebeldbnode01"}
}]
}
}}
Error:
{
"error": {
"root_cause": [
{
"type": "parsing_exception",
"reason": "[range] malformed query, expected [END_OBJECT] but found [FIELD_NAME]",
"line": 5,
"col": 5
}
],
"type": "parsing_exception",
"reason": "[range] malformed query, expected [END_OBJECT] but found [FIELD_NAME]",
"line": 5,
"col": 5
},
"status": 400
}
The objective is to find max of a numberic field fron an index for last 30 minutes of a specific node.
SY
Your query is not properly formatted, it should look like this instead.
POST /sbl_nmon2019.12.02/_search
{
"size": 0,
"query": {
"bool": {
"filter": [
{
"range": {
"#timestamp": {
"gte": "now-30m"
}
}
},
{
"match": {
"Server": "siebeldbnode01"
}
}
]
}
},
"aggs": {
"max_cpu": {
"max": {
"field": "cpu_consumed"
}
}
}
}
MUST attribute values should be separate object.
Correct format:
POST /sbl_nmon2019.12.02/_search?size=0
{
"query": {
"bool": {
"must": [
{
"match": {
"Server": "siebeldbnode01"
}
},
{
"range": {
"#timestamp": {
"gte": "now-30m"
}
}
}
]
},
"aggs": {
"max_cpu": {
"field": "cpu_consumed"
}
}
}
}
Wrong Format:
"must" : [{
"range":{"#timestamp":{"gte": "now-30m"}},
"aggs":{"max_cpu" : {"field":"cpu_consumed"}},
"match":{"Server" : "siebeldbnode01"}
}]

malformed query, expected END_OBJECT but found FIELD_NAME error in Kibana (Elastic Search)

I am running the following GET query within my Kibana Console and for some reason I am getting a error in the response window as follows :
// error
[match] malformed query, expected [END_OBJECT] but found [FIELD_NAME]
Can anyone suggest why I am not able to use multiple match blocks within the 'should' section?
// response - if i take out one of the match blocks it works??
{
"error": {
"root_cause": [
{
"type": "parsing_exception",
"reason": "[match] malformed query, expected [END_OBJECT] but found [FIELD_NAME]",
"line": 9,
"col": 13
}
],
"type": "parsing_exception",
"reason": "[match] malformed query, expected [END_OBJECT] but found [FIELD_NAME]",
"line": 9,
"col": 13
},
"status": 400
}
// my query
GET _search
{
"query": {
"bool": {
"should": [
{
"match": {
"text": "facebook advice"
},
"match": {
"profile": "facebook advice"
}
}
],
"minimum_number_should_match": 1,
"filter": {
"term": {
"accountid": "22"
}
}
}
}
Your query is malformed. Write it like this instead:
GET _search
{
"query": {
"bool": {
"should": [
{
"match": {
"text": "facebook advice"
}
},
{
"match": {
"profile": "facebook advice"
}
}
],
"minimum_number_should_match": 1,
"filter": {
"term": {
"accountid": "22"
}
}
}
}
}
Give the below query a try.. It works for me.
-------- working console query -------------
POST /usage-metering-stats/_search?size=10
{
"query": {
"bool": {
"must": [{
"term": {
"tenantId": "2222"
}
},
{
"term": {
"instanceId": "1212"
}
},
{
"term": {
"cspId": "25680"
}
},
{
"term": {
"api": "2"
}
}
]
}
},
"aggs": {
"totalCount": { "sum": { "field": "count" } }
}
}

Elasticsearch (version 2.3) Function Score Query with filtered type query

I am very new to elastic search, We are migrating from Solr to elastic-search. As part of migration working converting existing Solr query to elastic-search DSL query.
Here is the DSL query I have partially completed using function score feature.
{
"query": {
"function_score": {
"query": {
"filtered": {
"match": {
"name": "barack obama"
},
"filter": {
"range": {
"relevance": {
"gte": 6
}
},
"bool": {
"must_not": [
{
"terms": {
"classIds": [
199,
220
],
"execution": "and"
}
}
],
"must": [
{
"term": {
"classIds": 10597
}
}
]
}
}
}
},
"boost_mode": "replace",
"functions": [
{
"script_score": {
"script": {
"lang": "groovy",
"file": "calculate-score",
"params": {
"relevance_boost": 1,
"class_penalize": 0.25
}
}
}
}
]
}
}
}
This query returning error while am running against elastic-search cluster. Please help me to figure out the issue.
Here calculate-score is groovy script and its working fine, I tested that with simple query.
Here is the error response:
{
"error": {
"root_cause": [
{
"type": "query_parsing_exception",
"reason": "[filtered] query does not support [match]",
"index": "nodes_5e27a7d3-b370-40bd-9e71-cf04a36297c0",
"line": 6,
"col": 11
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "nodes_5e27a7d3-b370-40bd-9e71-cf04a36297c0",
"node": "NOAwAtVwQS25egu7AIaHEg",
"reason": {
"type": "query_parsing_exception",
"reason": "[filtered] query does not support [match]",
"index": "nodes_5e27a7d3-b370-40bd-9e71-cf04a36297c0",
"line": 6,
"col": 11
}
}
]
},
"status": 400
}
Here is Solr query I am trying to convert to elastic-search:
SOLR QUERY (UNIQUE_NODE_CORE): q={!boost b="product(pow(field(relevance),1.0000),if(exists(query({!v='all_class_ids:226'})),0.25,1),if(exists(query({!v='all_class_ids:14106'})),0.25,1),if(exists(query({!v='all_class_ids:656'})),0.25,1))"}
raw_name:"barack obama"
&rows=1
&start=0
&sort=score desc,relevance desc
-&fq=class_id:"10597"
-fq=relevance:[6 TO *]
-&fq=-all_class_ids:"14127"
-&fq=-all_class_ids:"14106"
-&fq=-all_class_ids:"226"
&fl=ontology_id,url_friendly_name,name,score,raw_notable_for,property_207578
Just need help to run filtered query with function score.
Great job, you're almost there, you're just missing a query section inside your filtered query in order to wrap the match query. As well, the range filter can be inserted into the bool/must. Quite a mouthful, I know.
{
"query": {
"function_score": {
"query": {
"filtered": {
"query": {
"match": {
"name": "barack obama"
}
},
"filter": {
"bool": {
"must_not": [
{
"terms": {
"classIds": [
199,
220
],
"execution": "and"
}
}
],
"must": [
{
"range": {
"relevance": {
"gte": 6
}
}
},
{
"term": {
"classIds": 10597
}
}
]
}
}
}
},
"boost_mode": "replace",
"functions": [
{
"script_score": {
"script": {
"lang": "groovy",
"file": "calculate-score",
"params": {
"relevance_boost": 1,
"class_penalize": 0.25
}
}
}
}
]
}
}
}
Note that since ES 2.0 the filtered query is deprecated and you can rewrite it with a bool/must/filter query like this:
{
"query": {
"function_score": {
"query": {
"bool": {
"must": {
"match": {
"name": "barack obama"
}
},
"filter": [
{
"range": {
"relevance": {
"gte": 6
}
}
},
{
"term": {
"classIds": 10597
}
}
],
"must_not": [
{
"terms": {
"classIds": [
199,
220
],
"execution": "and"
}
}
]
}
},
"boost_mode": "replace",
"functions": [
{
"script_score": {
"script": {
"lang": "groovy",
"file": "calculate-score",
"params": {
"relevance_boost": 1,
"class_penalize": 0.25
}
}
}
}
]
}
}
}

Elasticsearch: Query nested object contained within an object

I'm struggling to build a query where I can do a nested search across a sub-object of a document.
Say I have the following index/mapping:
curl -XPOST "http://localhost:9200/author/" -d '
{
"mappings": {
"item": {
"properties": {
"books": {
"type": "object",
"properties": {
"data": {
"type": "nested"
}
}
}
}
}
}
}
'
And the following 2 documents in the index:
{
"id": 1,
"name": "Robert Louis Stevenson",
"books": {
"count": 2,
"data": [
{
"id": 1,
"label": "Treasure Island"
},
{
"id": 3,
"label": "Dr Jekyll and Mr Hyde"
}
]
}
}
and
{
"id": 2,
"name": "Philip K. Dick",
"books": {
"count": 1,
"data": [
{
"id": 4,
"label": "Do Android Dream of Electric Sheep"
}
]
}
}
I have an array of Book ID's, say [1,4]; how would I write a query which does a keyword search of the author name AND only returns them if they wrote one of the books in the array?
I haven't managed to get a query which doesn't cause some sort of query parse_exception, but as a starting block, here's the current iteration of my query - maybe it's obvious where I'm going wrong?
{
"query": {
"bool": {
"must": {
"match": {
"label": "Louis"
}
}
},
"nested": {
"path": "books.data",
"query": {
"bool": {
"must": {
"terms": {
"books.data.id": [
1,
4
]
}
}
}
}
}
},
"from": 0,
"size": 8
}
In the above scenario I'd like the document for Mr Robert Louis Stevenson to be returned, as his name contains Louis and he wrote book ID 1.
For what it's worth, the current error I get looks like this:
{
"error": {
"root_cause": [
{
"type": "parse_exception",
"reason": "failed to parse search source. expected field name but got [START_OBJECT]"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "author",
"node": "sCk3su4YSnqhvdTGjOztlw",
"reason": {
"type": "parse_exception",
"reason": "failed to parse search source. expected field name but got [START_OBJECT]"
}
}
]
},
"status": 400
}
This makes me feel like I've got my "nested" object all wrong, but the docs suggest that I'm right!
You have it almost right, the nested query must simply be located inside the bool one like in the query below. Also the match query needs to be made on the name field since this is where the author name is stored:
{
"query": {
"bool": {
"must": [
{
"match": {
"name": "Louis"
}
},
{
"nested": {
"path": "books.data",
"query": {
"bool": {
"must": {
"terms": {
"books.data.id": [
1,
4
]
}
}
}
}
}
}
]
}
},
"from": 0,
"size": 8
}

Resources