Date Range Query using Search Template in Elasticsearch - elasticsearch

We are facing issue while framing date range query using search template in Elasticsearch. It is working fine, with one conditional clause, but when multiple conditions are provided, we are getting following error.
{
"script": {
"lang": "mustache",
"source": "{
\"query\":{
\"bool\":{
\"must\":[
{{#since}}
{\"range\":
{\"#timestamp\":
{
{{#from}}\"from\":\"{{from}}\"{{/from}}
}
}
},{{/since}}
{\"query_string\":
{
\"query\":\"(title:({{query_string}}))\"
}
}
]
}
}
}"
}
}
Error:
{
error: {
root_cause: [
{
type: "general_script_exception",
reason: "Failed to compile stored script [dateTemplate] using lang [mustache]",
}
],
type: "general_script_exception",
reason: "Failed to compile stored script [dateTemplate] using lang [mustache]",
caused_by: {
type: "mustache_exception",
reason: "Improperly closed variable in query-template:1",
},
},
status: 500,
}
Query:
{
"id": "dateTemplate",
"params": {
"query_string": "*"
}
}
Same is working fine for this Template:
{
"script": {
"lang": "mustache",
"source": "{\"query\":{\"bool\":{\"must\":[{{#since}}{\"range\": {\"#timestamp\": {\"from\": \"{{since}}\"}}},{{/since}}{\"query_string\":{\"query\":\"(title:({{query_string}}))\"}}]}}}"
}
}
Query
{
"id": "date",
"params": {
"query_string": "*",
"since": "2018-07-23"
}
}

First of all, I suggest you rewrite your template using triple quotes, as it's much easier to read and maintain, like this:
POST _scripts/dateTemplate
{
"script": {
"lang": "mustache",
"source": """
{
"query": {
"bool": {
"must": [
{{#since}}
{
"range": {
"#timestamp": {
{{#from}}"from": "{{from}}"{{/from}}
}
}
},
{{/since}}
{
"query_string": {
"query": "(title:({{query_string}}))"
}
}
]
}
}
}
"""
}
}
Then, the correct way to invoke that query is like this (i.e. you're missing the from variable in your params object):
{
"id": "dateTemplate",
"params": {
"query_string": "*",
"since": {
"from": "2018-07-23"
}
}
}

Related

How do I get the size of a 'nested' type array through a Painless script in Elasticsearch version 6.7?

I am using Elasticsearch version 6.7. I have the following mapping:
{
"customers": {
"mappings": {
"customer": {
"properties": {
"name": {
"type": "keyword"
},
"permissions": {
"type": "nested",
"properties": {
"entityId": {
"type": "keyword"
},
"entityType": {
"type": "keyword"
},
"permission": {
"type": "keyword"
},
"permissionLevel": {
"type": "keyword"
},
"userId": {
"type": "keyword"
}
}
}
}
}
}
}
}
I want to run a query to that shows all customers who have > 0 permissions. I have tried the following:
{
"query": {
"bool": {
"filter": {
"script": {
"script": {
"lang": "painless",
"source": "params._source != null && params._source.permissions != null && params._source.permissions.size() > 0"
}
}
}
}
}
}
But this returns no hits because params._source is null as Painless does not have access to the _source document according to this Stackoverflow post. How can I write a Painless script that gives me all customers who have > 0 permissions?
Solution 1: Using Script with must query
POST <your_index_name>/_search
{
"query": {
"bool": {
"must": [
{
"script": {
"script": {
"lang": "painless",
"inline": """
ArrayList st = params._source.permissions;
if(st!=null && st.size()>0)
return true;
"""
}
}
}
]
}
}
}
Solution 2: Using Exists Query on nested fields
You could simply make use of Exists query something like the below to get customers who have > 0 permissions.
Query:
POST <your_index_name>/_search
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "permissions",
"query": {
"bool": {
"should": [
{
"exists":{
"field": "permissions.permission"
}
},
{
"exists":{
"field": "permissions.entityId"
}
},
{
"exists":{
"field": "permissions.entityType"
}
},
{
"exists":{
"field": "permissions.permissionLevel"
}
}
]
}
}
}
}]
}
}
}
Solution 3: Create definitive structure but add empty values to the fields
Another alternative would be to ensure all documents would have the fields.
Basically,
Ensure that all the documents would have the permissions nested document
However for those who would not have the permissions, just set the field permissions.permission to 0
Construct a query that could help you get such documents accordingly
Below would be a sample document for a user who doesn't have permissions:
POST mycustomers/customer/1
{
"name": "john doe",
"permissions": [
{
"entityId" : "null",
"entityType": "null",
"permissionLevel": 0,
"permission": 0
}
]
}
The query in that case would be as simple as this:
POST <your_index_name>/_search
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "permissions",
"query": {
"range": {
"permissions.permission": {
"gte": 1
}
}
}
}
}
]
}
}
}
Hope this helps!

how to fetch last 30 minutes records from elastic search

I am using following query to fetch all last 30 minutes records using elastic search, but I'm getting parsable error on line "now-30m".
Query:
{
"query": {
"bool": {
"must": [
{
"match": {
"appName": "magnus-alerting-system"
}
},
{
"match": {
"countryCode": "US2"
}
},
{
"match": {
"eventCode": 201
}
},
{
"match": {
"extractName": "LaborDemand"
}
},{
"range": {
"eventPostTimestamp": {
**"gte": "now()-30m"**
}
}
}
]
}
}
}
Error on Postman while executing service:
"root_cause": [
{
"type": "number_format_exception",
"reason": "For input string: \"now()-30m\""
}
]
PLease let me know how to correct it.
The reason is because now()-30m in elasticsearch is wrong since the correct format is just "now".
Documentation
Hence the correct query is the following:
{
"query": {
"bool": {
"must": [
{
"match": {
"appName": "magnus-alerting-system"
}
},
{
"match": {
"countryCode": "US2"
}
},
{
"match": {
"eventCode": 201
}
},
{
"match": {
"extractName": "LaborDemand"
}
},{
"range": {
"eventPostTimestamp": {
"gte": "now-30m"
}
}
}
]
}
}
}
The correct syntax for using data math in range query for date field would be as below:
{
"range": {
"eventPostTimestamp": {
"gte": "now-30m"
}
}
}

Elasticsearch search template not working with terms query

I have an index with a long field and i am just trying to use search template to use terms query but it throws exception.
"pid": {
"type": "long"
}
Search Template:
PUT /_search/template/article_query_template
{
"template": {
"query": {
"terms": {
"pid": "{{articleId}}"
}
}
}
}
Search Query :
POST test2*/_search
{
"query": {
"template": {
"id": "article_query_template",
"params" : {
"articleId" : ["1"]
}
}
}
}
Exception : reason": "[terms] query does not support [pid]".
Its working without template. How to fix this issue.
Create your template like :
{
"source": {
"query": {
"term": {
"message": "{{query_string}}"
}
}
}
}
and pass params like :
{
"id": "<templateName>",
"params": {
"query_string": "search for these words"
}}
More details are available here.

Is it possible to update nested field by query?

I am using update by query plugin (https://github.com/yakaz/elasticsearch-action-updatebyquery/) to update documents by query.
In my case, there is nested field in document, the mapping is something like this:
"mappings": {
"mytype": {
"properties": {
"Myfield1": {
"type": "nested",
"properties": {
"field1": {
"type": "string"
},
"field2": {
"type": "long"
}
}
},
"Title": {
"type": "string"
}
}
}
}
Then I want to update the nested field Myfield1 by query with following request:
But unfortunately, it does not work.
{
"query": {
"match": {
"Title": "elasticsearch"
}
},
"script": "ctx._source.Myfield1 = [{'nestfield1':'foo blabla...','nestfield2':100},{'nestfield1':'abc...','nestfield2':200}]"
}
Does update by query support nested object?
BTW: any other ways to update document by query?
Is the update by query plugin the only choice?
This example uses _update_by_query
POST indexname/type/_update_by_query
{
"query": {
"match": {
"Title": "elasticsearch"
}
},
"script": {
"source": "ctx._source.Myfield1= params.mifieldAsParam",
"params": {
"mifieldAsParam": [
{
"nestfield1": "foo blabla...",
"nestfield2": 100
},
{
"nestfield1": "abc...",
"nestfield2": 200
}
]
},
"lang": "painless"
}
}
Nested elements need to be iterated in painless script to update values
POST /index/_update_by_query
{
"script": {
"source": "for(int i=0;i<=ctx._source['Myfield1'].size()-1;i++){ctx._source.Myfield1[i].field1='foo blabla...';ctx._source.Myfield1[i].field2=100}",
"lang": "painless"
},
"query": {
"match": {
"Title": "elasticsearch"
}
}
}
Nested elements value update if index is known
POST /index/_update_by_query
{
"script": {
"source": "ctx._source.Myfield1[0].field1='foo blabla...';ctx._source.Myfield1[0].field2=100;ctx._source.Myfield1[1].field1='abc...';ctx._source.Myfield1[1].field2=200;",
"lang": "painless"
},
"query": {
"match": {
"Title": "elasticsearch"
}
}
}
You can try with params, something like this:
"query" : {
"match_all" : {}
},
"script" : "ctx._source.Myfield1 = Myfield1;",
"params": {
"Myfield1": {
"nestfield1": "foo blabla..."
}
}
In my case I'm moving the data from not nested fields in nested fields. I need to add fake information to initialize the nested field. It looks like that:
"query" : {
"match_all" : {}
},
"script" : "ctx._source.Myfield1 = Myfield1; ctx._source.Myfield1.nestfield1 = ctx._source.Myfield1Nestfield1; ctx._source.Myfield1.nestfield2 = ctx._source.Myfield1Nestfield2;",
"params": {
"Myfield1": {
"nestfield1": "init_data"
}
}

ElasticSeach query not return results

I got 2 querys, the different between them is just 1 filter term.
The first query:
GET _search
{
"query": {
"filtered": {
"query": {
"query_string": {
"query": "*"
}
},
"filter": {
"and": {
"filters": [
{
"term": {
"type": "log"
}
},
{
"term": {
"context.blueprint_id": "adv1"
}
},
{
"term": {
"context.deployment_id": "deploy1"
}
}
]
}
}
}
}
}
return this result:
{
"_source": {
"level": "info",
"timestamp": "2014-03-24 10:12:41.925680",
"message_code": null,
"context": {
"blueprint_id": "Adv1",
"execution_id": "27efcba7-3a60-4270-bbe2-9f17d602dcef",
"deployment_id": "deploy1"
},
"type": "log",
"#version": "1",
"#timestamp": "2014-03-24T10:12:41.927Z"
}
}
The second query is:
{
"query": {
"filtered": {
"query": {
"query_string": {
"query": "*"
}
},
"filter": {
"and": {
"filters": [
{
"term": {
"type": "log"
}
},
{
"term": {
"context.blueprint_id": "adv1"
}
},
{
"term": {
"context.deployment_id": "deploy1"
}
},
{
"term": {
"context.execution_id": "27efcba7-3a60-4270-bbe2-9f17d602dcef"
}
}
]
}
}
}
}
}
return empty results.
the different between them is in the second query, i just add this term:
{
"term": {
"context.execution_id": "27efcba7-3a60-4270-bbe2-9f17d602dcef"
}
}
and in the result we can see that there is result match to that query, but it still not work.
what i'm doing wrong here?
thanks.
By default, ElasticSearch will treat string fields as text and will analyze them (i.e. tokenize, stem etc. before indexing). This means you might not be able to find them when searching for their exact content.
You should make sure the mapping for the execution_id field is not analyzed. Start with GET /_mappings and work from there. :)

Resources