Check null parameter in painless - Elasticsearch - elasticsearch

How are we supposed to check whether a key in map passed as parameter to a painless script has a value? I am currently doing this in Elasticsearch 6.8.4
if (params.feedId!=null) {
whatever()
}
but it throws this exception when
params.feedId
is null
ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Cannot cast null to a primitive type [void].]]
It works fine with Elasticsearch 5.6.13, so there seems to be a breaking change but I couldn't find anything so far.
UPDATE: It was actually not a problem with this part of the script but in some parts of it we were doing return null; to break the process of the script, which does not seem to be working anymore for some reason. To give more context, I am doing those updates using the Java API, more specifically the Rest High Level client.
I have been able to reproduce the issue using a curl command like this
curl -X POST "localhost:9200/myindex/_doc/1/_update?pretty" -H 'Content-Type: application/json' -d'
{
"script": {
"source": "ctx._source.result = true; return null;"
}
}'
This produces the next output:
{
"error" : {
"root_cause" : [
{
"type" : "remote_transport_exception",
"reason" : "[sBix--f][172.18.0.4:9300][indices:data/write/update[s]]"
}
],
"type" : "illegal_argument_exception",
"reason" : "failed to execute script",
"caused_by" : {
"type" : "script_exception",
"reason" : "compile error",
"script_stack" : [
"... ce.result = true; return null;",
" ^---- HERE"
],
"script" : "ctx._source.result = true; return null;",
"lang" : "painless",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "Cannot cast null to a primitive type [void]."
}
}
},
"status" : 400
}
Just replacing return null; for return; makes this script working fine in ES 6.8.4.
Found this issue at Elasticsearch: https://github.com/elastic/elasticsearch/issues/35888. There are some inconsistencies in the way short-circuit works in different versions of Elasticsearch.

Related

Getting a timestamp exception when I try to update an unrelated field using painless in elasticsearch

Im trying to run the following script
POST /data_hip/_update/1638643727.0
{
"script":{
"source":"ctx._source.avgmer=4;"
}
}
But I am getting the following error.
{
"error" : {
"root_cause" : [
{
"type" : "mapper_parsing_exception",
"reason" : "failed to parse field [#timestamp] of type [date] in document with id '1638643727.0'. Preview of field's value: '1.638642742E12'"
}
],
"type" : "mapper_parsing_exception",
"reason" : "failed to parse field [#timestamp] of type [date] in document with id '1638643727.0'. Preview of field's value: '1.638642742E12'",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "failed to parse date field [1.638642742E12] with format [epoch_millis]",
"caused_by" : {
"type" : "date_time_parse_exception",
"reason" : "Failed to parse with all enclosed parsers"
}
}
},
"status" : 400
}
this is strange because on queries (not updates) the date time is parsed fine.
The timestamp field mapping is as follows
"#timestamp": {
"type":"date",
"format":"epoch_millis"
},
I am running elasticsearch 7+
EDIT:
Adding my index settings
{
"data_hip" : {
"settings" : {
"index" : {
"routing" : {
"allocation" : {
"include" : {
"_tier_preference" : "data_content"
}
}
},
"number_of_shards" : "1",
"provided_name" : "data_hip",
"creation_date" : "1638559533343",
"number_of_replicas" : "1",
"uuid" : "CHjkvSdhSgySLioCju9NqQ",
"version" : {
"created" : "7150199"
}
}
}
}
}
Im not running an ingest pipeline
The problem is the scientific notation, the 'E12' suffix, being in a field that ES is expecting to be an integer.
Using this reprex:
PUT so_test
{
"mappings": {
"properties": {
"ts": {
"type": "date",
"format": "epoch_millis"
}
}
}
}
# this works
POST so_test/_doc/
{
"ts" : "123456789"
}
# this does not, throws the same error you have IRL
POST so_test/_doc/
{
"ts" : "123456789E12"
}
I'm not sure how/where those values are creeping in, but they are there in the document you are passing to ES.

Is "Number" restricted word in elasticsearch painless?

I'm working with elasticsearch painless more than two years.
These days I found out a weird problem with naming painless variable as "Number". When "Number" (first letter is uppercase) is used in painless script, error occured.
I sucessfully reproduce this scenario.
I'm using elasticsearch version 6.8.
curl -X POST "http://localhost:9201/logs-my_app-default/_search?size=2&pretty" -H 'Content-Type: application/json' -d '{"_source": ["-1"], "script_fields": {"SCRIPT_FLAG": {"script": {"lang": "painless", "source": " def Number = 0; "}}}}'
The problem comes from:
def Number = 0;
Thrown error is:
"error" : {
"root_cause" : [
{
"type" : "script_exception",
"reason" : "compile error",
"script_stack" : [
" def Number = 0; ",
" ^---- HERE"
],
"script" : " def Number = 0; ",
"lang" : "painless"
}
],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [
{
"shard" : 0,
"index" : "logs-my_app-default",
"node" : "9jBNNKjdSIO6I8UH_HLVRw",
"reason" : {
"type" : "script_exception",
"reason" : "compile error",
"script_stack" : [
" def Number = 0; ",
" ^---- HERE"
],
"script" : " def Number = 0; ",
"lang" : "painless",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "invalid sequence of tokens near ['Number'].",
"caused_by" : {
"type" : "no_viable_alt_exception",
"reason" : null
}
}
}
}
]
},
"status" : 500
}
This is happening on any elasticsearch index for me, es version 6.8
To be more precise, this is the index used in scenario shown here.
curl -X POST "localhost:9201/logs-my_app-default/_doc?pretty" -H 'Content-Type: application/json' -d'
{
"#timestamp": "2099-05-06T16:21:15.000Z",
"event": {
"original": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] \"GET /images/bg.jpg HTTP/1.0\" 200 24736"
}
}
'
Mapping as well:
curl -X GET "localhost:9201/logs-my_app-default/_mapping?pretty"
{
"logs-my_app-default" : {
"mappings" : {
"_doc" : {
"properties" : {
"#timestamp" : {
"type" : "date"
},
"event" : {
"properties" : {
"original" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}
}
}
}
Any variant of string "Number" as "number" (all lowercase) works.
"Numb", "Numbe" works as well..
Any help?
Thank you.
SOLUTION:
Almost all of the casesensitive keywords mentioned here are throwing the same error as shown above.
Good practice for me is to use underscore as prefix when creating painless variable with same name as elasticsearch field ..
For example:
def _Number = 0; // instead of def Number = 0,
def _Void = 0; // instead of def Void = 0; ,
def _System = 0; // instead of def System = 0; ..
The Painless scripting language is very similar to the Java programming language, compiles directly into JVM bytecode, and its API borrows a lot from the Java API, namely the Number class.
For this reason, you cannot use Number as an arbitrary variable identifier.

'Unknown BaseAggregationBuilder [composite] error' when running elasticsearch composite aggregation

I'm trying to create a composite aggregation per the documentation here:
https://www.elastic.co/guide/en/elasticsearch/reference/6.8/search-aggregations-bucket-composite-aggregation.html
I'm basically following this example:
curl -X GET "localhost:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
"aggs" : {
"my_buckets": {
"composite" : {
"sources" : [
{ "product": { "terms" : { "field": "product" } } }
]
}
}
}
}
'
but every time I try to run the code I get the below error regardless of which field I try to aggregate on:
{
"error" : {
"root_cause" : [
{
"type" : "unknown_named_object_exception",
"reason" : "Unknown BaseAggregationBuilder [composite]",
"line" : 5,
"col" : 27
}
],
"type" : "unknown_named_object_exception",
"reason" : "Unknown BaseAggregationBuilder [composite]",
"line" : 5,
"col" : 27
},
"status" : 400
}
I did some digging around and haven't seen the error 'Unknown BaseAggregationBuilder [composite]' come up anywhere else so I thought I'd post this question here to see if anyone has run into a similar issue. Cardinality and regular terms aggregation work fine. Also to clarify, I'm running on v6.8
Composite aggs were released in 6.1.0. The error sounds like you cannot possibly be using >=6.1 but some older ver.
What's your version.number when you run curl -X GET "localhost:9200"?

Elasticsearch not allowing brackets for fields name for scripting

I've tried to use update api. I've inserted a document which contains a list.
INSERT:
curl -XPOST "http://localhost:9200/t/t/1" -d'
{
"hobbies(first)" : ["a", "b"]
}'
UPDATE QUERY:
curl -XPOST localhost:9200/t/t/1/_update?pretty -d '{ "script" : {
"inline": "ctx._source.hobbies(first).add(params.new_hobby)",
"params" : {
"new_hobby" : "c"
}
}
}'
ERROR:
{
"error" : {
"root_cause" : [
{
"type" : "remote_transport_exception",
"reason" : "[aaBiwwv][172.17.0.2:9300][indices:data/write/update[s]]"
}
],
"type" : "illegal_argument_exception",
"reason" : "failed to execute script",
"caused_by" : {
"type" : "script_exception",
"reason" : "compile error",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "Variable [first] is not defined."
},
"script_stack" : [
"ctx._source.hobbies(first).add(params.new_hob ...",
" ^---- HERE"
],
"script" : "ctx._source.hobbies(first).add(params.new_hobby)",
"lang" : "painless"
}
},
"status" : 400
}
When I've tried to update list, I've got error above. I've realized that when I removed part with brackets('(first)') from my field's name, it's working. How can I prepare an update query with a field name with brackets?
Thanks in advance.
this is a horrible convention for field names, just stick with alphanumerics (please keep in mind, that someone after you has to maintain this, so it would be so much nicer to work with cleaner data in Elasticsearch). You can try ctx.source['hobbies(first)']

elasticsearch field mapping affects acorss different types in same index

I was told that "Every type has its own mapping, or schema definition" at the official guide.
But the fact I've met is the mapping can affect other types within the same index. Here is the situation:
Mapping definition:
[root#localhost agent]# curl localhost:9200/agent*/_mapping?pretty
{
"agent_data" : {
"mappings" : {
"host" : {
"_all" : {
"enabled" : false
},
"properties" : {
"ip" : {
"type" : "ip"
},
"node" : {
"type" : "string",
"index" : "not_analyzed"
}
}
},
"vul" : {
"_all" : {
"enabled" : false
}
}
}
}
}
and then I index a record:
[root#localhost agent]# curl -XPOST 'http://localhost:9200/agent_data/vul?pretty' -d '{"ip": "1.1.1.1"}'
{
"error" : {
"root_cause" : [ {
"type" : "mapper_parsing_exception",
"reason" : "failed to parse [ip]"
} ],
"type" : "mapper_parsing_exception",
"reason" : "failed to parse [ip]",
"caused_by" : {
"type" : "number_format_exception",
"reason" : "For input string: \"1.1.1.1\""
}
},
"status" : 400
}
Seems that it tries to parse the ip as a number. So I put a number in this field:
[root#localhost agent]# curl -XPOST 'http://localhost:9200/agent_data/vul?pretty' -d '{"ip": "1123"}'
{
"error" : {
"root_cause" : [ {
"type" : "remote_transport_exception",
"reason" : "[Argus][127.0.0.1:9300][indices:data/write/index[p]]"
} ],
"type" : "illegal_argument_exception",
"reason" : "mapper [ip] cannot be changed from type [ip] to [long]"
},
"status" : 400
}
This problem goes away if I explicitly define the ip field of vul type as ip field-type.
I don't quite understand the behavior above. Do I miss something?
Thanks in advance.
The statement
Every type has its own mapping, or schema definition
is true. But this is not complete information.
There may be conflicts between different types with the same field within one index.
Mapping - field conflicts
Mapping types are used to group fields, but the fields in each
mapping type are not independent of each other. Fields with:
the same name
in the same index
in different mapping types
map to the same field internally, and must have the same mapping. If a
title field exists in both the user and blogpost mapping types, the
title fields must have exactly the same mapping in each type. The only
exceptions to this rule are the copy_to, dynamic, enabled,
ignore_above, include_in_all, and properties parameters, which may
have different settings per field.

Resources