ElasticSearch Illegal list shortcut value [id] "Update By Query" - elasticsearch

I get Illegal list shortcut value [id]. trying to update this document with this query. What Am I missing
events" : { "type" : "nested" }
location: {"type" : "nested"} nested type of objects
id: {"type" : "text"}
POST event_lists/_update_by_query?conflicts=proceed
{
"script": {
"lang": "painless",
"source": """
for (int i=0; i< ctx._source.events.length; i++){
if(params.event_ids.contains(ctx._source.events[i].id)){
ctx._source.events[i].location = params.location;
break;
}
}
""",
"params": {
"event_ids": ["12345"],
"location": location_object
}
}
}
When trying to use Kibana to debug
Debug.explain(ctx._source.events[i].id);
I get
"root_cause": [
{
"type": "script_exception",
"reason": "runtime error",
"painless_class": "java.lang.String",
"to_string": "ETo3zoABiBlDN0geqAGN",
"java_class": "java.lang.String",
"script_stack": [
"Debug.explain(ctx._source.events[i].id); \n ",
" ^---- HERE"
]

I ended up doing a check if the object is a list vs. an object and it seems to be working.
for (event in ctx._source.events ){
if(event instanceof List && event.size() > 0) {
if(params.event_ids.contains(event[0].id)){
event[0].location = params.location;
break;
}
}else{
if(params.event_ids.contains(event.id)){
event.location = params.location;
break;
}
}
}

Related

Change field values using Elasticsearch Painless

I want to check the length of each field under the object alert.
If it's longer than X, change it to value was truncated since it was too long
"_source" : {
"random" : 123455,
},
"#timestamp" : "2021-10-15T21:55:12.938+0000",
"application" : "app",
"alert" : {
"app_key" : "XYZ",
"host" : "Hostname",
"check" : "CPU overloaded",
"status" : "ok"
},
For some reason, I get this error when trying to apply my script:
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"java.base/java.util.HashMap$HashIterator.nextNode(HashMap.java:1584)",
"java.base/java.util.HashMap$EntryIterator.next(HashMap.java:1617)",
"java.base/java.util.HashMap$EntryIterator.next(HashMap.java:1615)",
"for (alert in ctx._source.alert.entrySet())\n {\n ",
" ^---- HERE"
],
"script" : " ...",
"lang" : "painless",
"position" : {
"offset" : 38,
"start" : 7,
"end" : 65
},
"caused_by" : {
"type" : "concurrent_modification_exception",
"reason" : null
Script:
{
"script": {
"lang": "painless",
"source": """
for (alert in ctx._source.alert.entrySet())
{
if (alert.getValue().length() > params.number_of_characters) {
ctx._source.alert[alert] = "value was truncated since it was too long"
}
}
""",
"params": {
"number_of_characters": 5
}
}
}
You cannot modify the map directly while iterating over the set returned by entrySet, only through the setValue method of the entry.
Per the doc for entrySet():
Returns a Set view of the mappings contained in this map. The set is
backed by the map, so changes to the map are reflected in the set, and
vice-versa. If the map is modified while an iteration over the set is
in progress (except through the iterator's own remove operation, or
through the setValue operation on a map entry returned by the
iterator) the results of the iteration are undefined. The set supports
element removal, which removes the corresponding mapping from the map,
via the Iterator.remove, Set.remove, removeAll, retainAll and clear
operations. It does not support the add or addAll operations.
Try with the following script:
for (alert in ctx._source.alert.entrySet()){
if (alert.getValue().length() > params.number_of_characters) {
alert.setValue("value was truncated")
}
}
Eventually, with the help of Nicolas, I used this:
PUT _ingest/pipeline/rawpayload-ingest/
{
"processors": [
{
"script": {
"description": "Loop through all fields in alert object and shorten them if needed",
"lang": "painless",
"source": """
for (field in ctx.alert.entrySet()) {
if (field.getValue() instanceof String) {
if (field.getValue().length() > params.number_of_characters) {
field.setValue(field.getValue().substring(0, params.number_of_characters) + "...truncated")
}
}
}
""",
"params": {
"number_of_characters": 512
}
}
}
]
}

Root mapping definition has unsupported parameters(mapper_parsing_exception) while running elastic search painless script

{
"query": {
"match_all": {}
},
"script": {
"source":"if(ctx._source.containsKey('priority')){if (ctx._source.priority != null && ctx._source.priority != 'null') { ctx._source.priority_filter = String.valueOf(ctx._source.priority) } else { ctx._source.priority_filter = ''}}",
"lang": "painless"
}
}
I'm getting the below error when I try run the above script can anyone tell me where I'm going wrong and my elastic search version is v6.5.2
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "Root mapping definition has unsupported parameters: [query : {match_all={}}] [script : {source=if(ctx._source.containsKey('priority')){if (ctx._source.priority != null && ctx._source.priority != 'null') { ctx._source.priority_filter = String.valueOf(ctx._source.priority) } else { ctx._source.priority_filter = ''}}, lang=painless}]"
}
],
"type": "mapper_parsing_exception",
"reason": "Root mapping definition has unsupported parameters: [query : {match_all={}}] [script : {source=if(ctx._source.containsKey('priority')){if (ctx._source.priority != null && ctx._source.priority != 'null') { ctx._source.priority_filter = String.valueOf(ctx._source.priority) } else { ctx._source.priority_filter = ''}}, lang=painless}]"
},
"status": 400
}

Painless script to get time difference between two logs entries separated by a unique ID

I am trying to get time difference between two log entries like RequestExecuted and RequestReceived with a filed name MessageIdentifier. These valuse are linked by unique id named TransactionId. Below is my code to do the logic.
int timetaken=0;
int start=0;
String TransactionId;
int end=0;
for(int i = 0; i < 10; ++i){
if (doc['dissect.MessageIdentifier'].value[i]=='RequestReceived') {
start=params._source.dissect.timestamp[i];
TransactionId=params._source.dissect.TransactionId[i];
}
if( doc['dissect.MessageIdentifier'].value[i] =='RequestExecuted'
&& params._source.dissect.TransactionId == TransactionId) {
end=params._source.dissect.timestamp[i];
timetaken = end - start;
return timetaken;
}
}
When i compile my painless script it gives me an error:
lang": "painless",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Attempting to address a non-array-like type [java.lang.String] as an array."
Here is the index snippet:
Your help will be highly obliged.
Presuming your dissect field is an array of nested objects you can do the following:
Create index
PUT dissect
{
"mappings": {
"properties": {
"dissect" : {
"type": "nested",
"properties" : {
"MessageIdentifier" : {
"type" : "text",
"fielddata": true,
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"TransationId" : {
"type" : "text",
"fielddata": true,
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"timestamp" : {
"type" : "date"
}
}
}
}
}
}
Sync a sample
POST dissect/_doc
{
"dissect": [
{
"MessageIdentifier": "abc",
"timestamp": 200,
"TransationId": "xyz"
},
{
"MessageIdentifier": "RequestReceived",
"timestamp": 300,
"TransationId": "xyz"
},
{
"MessageIdentifier": "RequestExecuted",
"timestamp": 400,
"TransationId": "xyz"
}
]
}
Run your script field
GET dissect/_search
{
"script_fields": {
"timetaken": {
"script": {
"source": """
int timetaken = 0;
int start = 0;
String TransactionId;
int end = 0;
for (def dissect_item : params._source['dissect']) {
if (dissect_item['MessageIdentifier'] == 'RequestReceived') {
start = dissect_item['timestamp'];
TransactionId = dissect_item['TransactionId'];
}
if( dissect_item['MessageIdentifier'] =='RequestExecuted'
&& dissect_item['TransactionId'] == TransactionId) {
end = dissect_item['timestamp'];
timetaken = end - start;
return timetaken;
}
}
"""
}
}
}
}
yielding
[
{
"_index":"dissect",
"_type":"_doc",
"_id":"_v7u43EBW-D5QnrWmjtM",
"_score":1.0,
"fields":{
"timetaken":[
100 <-----
]
}
}
]
Key takeaway: you don't wanna be iterating over a hard-coded length of 10 but instead as for (def dissect_item : params._source['dissect'])

How does elasticsearch handle returns inside a scripted update query?

I can't find the relevant documentation describing the return keyword. Where is this documented?
I am running the following query
POST /myindex/mytype/FwOaGmQBdhLB1nuQhK1Q/_update
{
"script": {
"source": """
if (ctx._source.owner._id.equals(params.signedInUserId)){
for (int i = 0; i < ctx._source.managers.length; i++) {
if (ctx._source.managers[i].email.equals(params.managerEmail)) {
ctx._source.managers.remove(i);
return;
}
}
}
ctx.op = 'noop';
""",
"lang": "painless",
"params": {
"signedInUserId": "auth0|5a78c1ccebf64a46ecdd0d9c",
"managerEmail": "d#d.com"
}
},
"_source": true
}
but I'm getting the error
"type": "illegal_argument_exception",
"reason": "failed to execute script",
"caused_by": {
"type": "script_exception",
"reason": "compile error",
"script_stack": [
"... ve(i);\n return;\n }\n }\n ...",
" ^---- HERE"
],
"script": <the script here>,
"lang": "painless",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "invalid sequence of tokens near [';'].",
"caused_by": {
"type": "no_viable_alt_exception",
"reason": null
}
}
If I remove return keyword, then the script runs but I get the wrong behavior as expected. I can correct the behavior by using a Boolean to keep track of email removal, but why can't I return early?
It's hard to say, you could avoid null/void returns by passing a lambda comparator to either retainAll or removeIf
ctx._source.managers.removeIf(m -> m.email.equals(params.managerEmail))
Lambda expressions and method references work the same as Java’s.

elasticsearch kibana update document which has space using update by query

I have a field to update which has space in it.
POST /index/type/_update_by_query
{
"query": {
"match_phrase":{
"field": "value"
}
},
"script":{
"lang": "painless",
"inline": "ctx._source.Existing Field = New_Value"
}
}
But I get this error.
{
"error": {
"root_cause": [
{
"type": "script_exception",
"reason": "compile error",
"script_stack": [
"ctx._source.Existing Field = New_Value",
" ^---- HERE"
],
"script": "ctx._source.Existing Field = New_Value",
"lang": "painless"
}
],
"type": "script_exception",
"reason": "compile error",
"script_stack": [
"ctx._source.Existing Field = New_Value",
" ^---- HERE"
],
"script": "ctx._source.Existing Field = New_Value",
"lang": "painless",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "unexpected token ['Field'] was expecting one of [{<EOF>, ';'}]."
}
},
"status": 500
}
When I execute this query on a field which doesn't have space, it works fine.
How do I handle cases where there is a space in the field name?
ELK version = 5.4.3
I have read in the documentation that using spaces in field names is not advised, but these fields are dynamically created from a certain server and there are like 1M data entries every day. Hence I want to do a update_by_query on all the matching entries.
Try this one:
POST index/type/_update_by_query
{
"script":{
"lang": "painless",
"inline": "ctx._source['Existing Field'] = 'New Value'"
}
}
This is possible because ctx._source is an instance of painless Map, which is a normal Java HashMap. It allows you to access fields with weird characters and also add and remove fields in update queries.
Hope that helps!

Resources