Elasticsearch script - variable not defined - elasticsearch

I'm following an Elasticsearch tutorial, but ran into a problem when trying to use parameters in a script.
Step 1: Create a new document - OK (index = website; type = blog; id = 1)
curl -XPUT localhost:9200/website/blog/1?pretty -d '{
"title":"my first post",
"tags" : [ "tag1" ]
}'
Step 2: Use script to append an extra value to tags array - ERROR
curl -XPOST localhost:9200/website/blog/1/_update?pretty -d '{
"script" : "ctx._source.tags+=new_tag",
"params" : {
"new_tag" : "tag2"
}
}'
The error message is this, mentioning "reason" : "Variable [new_tag] is not defined." But I have defined the variable new_tag as described on the tutorial page. What am I doing wrong?
"error" : {
"root_cause" : [
{
"type" : "remote_transport_exception",
"reason" : "[mrukUvA][127.0.0.1:9300][indices:data/write/update[s]]"
}
],
"type" : "illegal_argument_exception",
"reason" : "failed to execute script",
"caused_by" : {
"type" : "script_exception",
"reason" : "compile error",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "Variable [new_tag] is not defined."
},
"script_stack" : [
"ctx._source.tags+=new_tag",
" ^---- HERE"
],
"script" : "ctx._source.tags+=new_tag",
"lang" : "painless"
}
},
"status" : 400
}
Step 2 (retry) Qualifying new_tag with params - ERROR
curl -XPOST localhost:9200/website/blog/1/_update?pretty -d '{
"script" : {
"inline": "ctx._source.tags+=params.new_tag",
"params" : {
"new_tag" : "tag2"
}
}
}'
Gives error
{
"error" : {
"root_cause" : [
{
"type" : "remote_transport_exception",
"reason" : "[mrukUvA][127.0.0.1:9300][indices:data/write/update[s]]"
}
],
"type" : "illegal_argument_exception",
"reason" : "failed to execute script",
"caused_by" : {
"type" : "script_exception",
"reason" : "runtime error",
"caused_by" : {
"type" : "class_cast_exception",
"reason" : "Cannot cast java.lang.String to java.util.ArrayList"
},
"script_stack" : [
"ctx._source.tags+=params.new_tag",
" ^---- HERE"
],
"script" : "ctx._source.tags+=params.new_tag",
"lang" : "painless"
}
},
"status" : 400
}
As a sanity check to make sure the document is valid
$ curl -XGET localhost:9200/website/blog/1?pretty
{
"_index" : "website",
"_type" : "blog",
"_id" : "1",
"_version" : 27,
"found" : true,
"_source" : {
"title" : "my first post",
"tags" : [
"tag1"
]
}
}
So the document does have the valid field tag which is an array.

Your syntax is slightly off, if you have parameters you need to inline the script. Try this:
curl -XPOST localhost:9200/website/blog/1/_update?pretty -d '{
"script" : {
"inline": "ctx._source.tags.add(params.new_tag)",
"params" : {
"new_tag" : "tag2"
}
}
}'

you can use "inline" although it is deprecated. now you alse can use "source" to replace "inline" for no warnings. for example:
"script" : {
"source": "ctx._source.tags.add(params.new_tag)",
"params": {
"new_tag":"tag1"
}
}

Related

Elastic Search update by query script for a string field

Unable to do Elastic Search update by query script for a string field. This is my query
curl -X POST "http://localhost:9200/z5-purchase-orders/_update_by_query?pretty" -H 'Content-Type: application/json' -d'
{
"script": {
"source": "ctx._source.facility_name = VCU",
"lang": "painless"
},
"query": {
"bool" : {
"must" : [
{ "match" : {"facility_id" : 0} },
{ "match" : {"customer_id" : 1002} }
]
}
}
}
'
I did the same query for integer field. It worked fine, with string its causing error
{
"error" : {
"root_cause" : [
{
"type" : "script_exception",
"reason" : "compile error",
"script_stack" : [
"... ._source.facility_name = VCU",
" ^---- HERE"
],
"script" : "ctx._source.facility_name = VCU",
"lang" : "painless"
}
],
"type" : "script_exception",
"reason" : "compile error",
"script_stack" : [
"... ._source.facility_name = VCU",
" ^---- HERE"
],
"script" : "ctx._source.facility_name = VCU",
"lang" : "painless",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "Variable [VCU] is not defined."
}
},
"status" : 400
}
Please help with this string update by query. I use elastic search version 7.
To fix that error, you simply need to stringify the value VCU
"ctx._source.facility_name = 'VCU'"
If you still get an error after that, it's going to be a different one, feel free to share it.
After escaping the characters it worked fine.
"script": {
"source": "ctx._source.facility_name = \"VCU\"",
"lang": "painless"
},

Cannot apply [*] operation to types [org.elasticsearch.index.fielddata.ScriptDocValues.Doubles] and [java.lang.Double]

Here is the mappeing of the attribute for the index-
"hoi" : {
"type" : "float"
}
This is the query that I want to execute. I want to change my attribute hoi with dynamic value and sort the doc on top of that, don't want to use runtime mapping as I have lower version of ES-
GET test/_search
{
"sort" : {
"_script" : {
"type" : "number",
"script" : {
"lang": "painless",
"source": "doc['hoi'] * params.factor",
"params" : {
"factor" : 1.1
}
},
"order" : "asc"
}
}
}
Error-
{
"error" : {
"root_cause" : [
{
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"doc['hoi'] * params.factor",
" ^---- HERE"
],
"script" : "doc['hoi'] * params.factor",
"lang" : "painless",
"position" : {
"offset" : 40,
"start" : 0,
"end" : 47
}
}
],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [
{
"shard" : 0,
"index" : "test",
"node" : "5sfrbDiLQDOg82_sneaU1g",
"reason" : {
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"doc['hoi'] * params.factor",
" ^---- HERE"
],
"script" : "doc['hoi'] * params.factor",
"lang" : "painless",
"position" : {
"offset" : 40,
"start" : 0,
"end" : 47
},
"caused_by" : {
"type" : "class_cast_exception",
"reason" : "Cannot apply [*] operation to types [org.elasticsearch.index.fielddata.ScriptDocValues.Doubles] and [java.lang.Double]."
}
}
}
]
},
"status" : 400
}
How to make it work?
doc['hoi'] itself is of type org.elasticsearch.index.fielddata.ScriptDocValues.Doubles and params.factor is of type java.lang.Double.
So the error you get means that some document has multiple values in your hoi field. So you need to reference it with doc['hoi'].value * params.factor instead, which will take the first hoi value from the array to compute the sort.
You can also check how many values there are with doc['hoi'].size().
UPDATE
If some docs don't have any values for hoi then you can adapt the script like this:
doc['hoi'].size() > 0 ? doc['hoi'].value * params.factor : 0

Elasticsearch - get docs with array field size greater than 1

I want to get all docs with a field, which is an array, with size greater than 1.
I'm using the following command:
curl -H "Content-Type: application/json" -XGET '127.0.0.1:9200/poly/_search?pretty' -d '
{ "query": { "bool": { "filter": { "script" : { "script" : "doc['emoji'].length > 1" } } } } }
'
but I get the following error:
{
"error" : {
"root_cause" : [
{
"type" : "script_exception",
"reason" : "compile error",
"script_stack" : [
"doc[emoji].length > 1",
" ^---- HERE"
],
"script" : "doc[emoji].length > 1",
"lang" : "painless",
"position" : {
"offset" : 4,
"start" : 0,
"end" : 21
}
}
],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [
{
"shard" : 0,
"index" : "poly",
"node" : "Dnp4BI4YSgCz-C-p8NmcTg",
"reason" : {
"type" : "query_shard_exception",
"reason" : "failed to create query: compile error",
"index_uuid" : "HlWRuJb5TY-L_2_9iyuVmg",
"index" : "poly",
"caused_by" : {
"type" : "script_exception",
"reason" : "compile error",
"script_stack" : [
"doc[emoji].length > 1",
" ^---- HERE"
],
"script" : "doc[emoji].length > 1",
"lang" : "painless",
"position" : {
"offset" : 4,
"start" : 0,
"end" : 21
},
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "cannot resolve symbol [emoji]"
}
}
}
}
],
"caused_by" : {
"type" : "script_exception",
"reason" : "compile error",
"script_stack" : [
"doc[emoji].length > 1",
" ^---- HERE"
],
"script" : "doc[emoji].length > 1",
"lang" : "painless",
"position" : {
"offset" : 4,
"start" : 0,
"end" : 21
},
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "cannot resolve symbol [emoji]"
}
}
},
"status" : 400
}
Nevertheless, the field "emoji" exists in my ElasticSearch docs, as you can see in the result to the following command:
curl -H "Content-Type: application/json" -XGET '127.0.0.1:9200/poly/_search?pretty' -d '
{
"_source": ["emoji"],
"query" : {
"constant_score" : {
"filter" : {
"exists" : {
"field" : "emoji"
}
}
}
}
}
'
Here is the result for the previous command:
"took" : 28,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 10000,
"relation" : "gte"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "poly",
"_type" : "_doc",
"_id" : "1307256887860174848",
"_score" : 1.0,
"_source" : {
"emoji" : [
"❤️"
]
}
},
{
"_index" : "poly",
"_type" : "_doc",
"_id" : "1278766523134414848",
"_score" : 1.0,
"_source" : {
"emoji" : [
"⏩"
]
}
},
{
"_index" : "poly",
"_type" : "_doc",
"_id" : "1298632385605431296",
"_score" : 1.0,
"_source" : {
"emoji" : [
"\uD83D\uDC47\uD83C\uDFFF",
"\uD83D\uDC47\uD83C\uDFFF",
"\uD83D\uDC47\uD83C\uDFFF",
"\uD83D\uDC47\uD83C\uDFFF"
]
}
},
{
"_index" : "poly",
"_type" : "_doc",
"_id" : "1300563120184651776",
"_score" : 1.0,
"_source" : {
"emoji" : [
"\uD83D\uDC4D",
"\uD83D\uDE00",
"\uD83D\uDE80"
]
}
},
]
}
}
Can someone tell me why I'm getting that error above?
You need to skip characters in your command check command below:
Script:
"script": {"script": "doc['\''emoji'\''].length > 1"
Command:
curl -H "Content-Type: application/json" -XGET '127.0.0.1:9200/poly/_search?pretty' -d '{ "query": {"bool": {"filter": {"script": {"script": "doc['\''emoji'\''].length > 1"}}}}}'

issue in creating a pipeline in elasticSearch

I'm trying to ingest a pipeline that contains grok, date and remove processors but i am getting a missing field error despite explicitly mentioning the field "message" under the docs
GET _ingest/pipeline/_simulate
{
"pipeline" : {
"processors" : [
{
"grok" : {
"field" : "message",
"pattern" : "%{COMMONAPACHELOG}"
}
},
{
"date" : {
"match_field" : "timestamp",
"match_formats" : ["dd/MMM/YYYY:HH:mm:ss Z"]
}
},
{
"remove" : {
"field" : "message"
}
}
]
},
"docs" : [
{
"_source" : {
"message" : "52.35.38.35 -- [19/Apr/2016:12:00:04 +0200] \"GET/ HTTP/1.1\" 200 24"
},
"_index" : "indexer"
}
]
}
and i'm Getting this Error please help
{
"error" : {
"root_cause" : [
{
"type" : "parse_exception",
"reason" : "[patterns] required property is missing",
"property_name" : "patterns",
"processor_type" : "grok",
"suppressed" : [
{
"type" : "parse_exception",
"reason" : "[field] required property is missing",
"property_name" : "field",
"processor_type" : "date"
}
]
}
],
"type" : "parse_exception",
"reason" : "[patterns] required property is missing",
"property_name" : "patterns",
"processor_type" : "grok",
"suppressed" : [
{
"type" : "parse_exception",
"reason" : "[field] required property is missing",
"property_name" : "field",
"processor_type" : "date"
}
]
},
"status" : 400
}
i tried to look for a video on youtube and i found someone with the same code and it executed well
here's the video
https://www.youtube.com/watch?v=PEHnBa19Gxs&t=1s
it's on minute 34
as it turns out that it worked at the youtube guy because it was on an older version.
this will work on the newer version
GET _ingest/pipeline/_simulate
{
"pipeline" : {
"processors" : [
{
"grok" : {
"field" : "message",
"patterns" : ["%{COMMONAPACHELOG}"]
}
},
{
"date" : {
"field" : "timestamp",
"formats" : ["dd/MMM/YYYY:HH:mm:ss Z"]
}
},
{
"remove" : {
"field" : "message"
}
}
]
},
"docs" : [
{
"_source" : {
"message" : "52.35.38.35 - - [19/Apr/2016:12:00:04 +0200] \"GET/ HTTP/1.1\" 200 24"
},
"_index" : "indexer"
}
]
}

Elasticsearch Updating field value issue

I am trying to work with update_by_query but cannot make it work.
Following is a simple query,
curl -X GET "172.17.0.3:9200/useripvsuserid/_search?pretty" -H 'Content-Type: application/json' -d'
{
"_source":"userid","query": {
"term": {
"userip": "10.0.30.181"
}
}
}
'
{
"took" : 3,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 10.803431,
"hits" : [
{
"_index" : "useripvsuserid",
"_type" : "_doc",
"_id" : "PhfBW3AB8mhGfmGvIs-j",
"_score" : 10.803431,
"_source" : {
"userid" : "hasan1855"
}
}
]
}
}
Following is the update_by_query that is not working. I am trying to replace userid value hasan1855 to arif. Where is the problem?
curl -X POST "172.17.0.3:9200/useripvsuserid/_update_by_query?pretty" -H 'Content-Type: application/json' -d'
{
"script": {
"source": "ctx._source.userid='arif';",
"lang": "painless"
},
"query": {
"term": {
"userip": "10.0.30.181"
}
}
}
'
{
"error" : {
"root_cause" : [
{
"type" : "script_exception",
"reason" : "compile error",
"script_stack" : [
"ctx._source.userid=arif;",
" ^---- HERE"
],
"script" : "ctx._source.userid=arif;",
"lang" : "painless"
}
],
"type" : "script_exception",
"reason" : "compile error",
"script_stack" : [
"ctx._source.userid=arif;",
" ^---- HERE"
],
"script" : "ctx._source.userid=arif;",
"lang" : "painless",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "Variable [arif] is not defined."
}
},
"status" : 400
}
It's the same issue as described here, i.e. the single quotes around arif conflict with the single quotes around the JSON query.
So you can either send your query in binary mode as explained in the link above, or escape the quotes, like this:
curl -X POST "172.17.0.3:9200/useripvsuserid/_update_by_query?pretty" -H 'Content-Type: application/json' -d'
{
"script": {
"source": "ctx._source.userid = \"arif\";", <---- escape quotes
"lang": "painless"
},
"query": {
"term": {
"userip": "10.0.30.181"
}
}
}
'

Resources